Context engineering: what fits and what gets dropped

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

"Context engineering" showed up this year as the term for something teams were already doing badly: deciding what goes into the model's input on a given call. The name is an upgrade over "prompt engineering" because it's honest about the object — it's not one clever string, it's a window that fills up over a run and has to be actively managed — but the term is getting used as a synonym for "write a good system prompt," which throws away the part that's actually hard. Writing a good system prompt is a one-time decision. Context engineering is a decision you make every turn, under a budget, about a working set that keeps changing shape while the run is still happening and you don't yet know which of today's facts will matter on turn eighty.

That's the discipline this post is actually about: not the prompt, the packing problem — what fits, what gets dropped, and what dropping the wrong thing costs you. I've written about pieces of this before from different angles. This is the synthesis: the constraint, the decision, the price, and the practice.

The physical constraint, and what it isn't

Start with what a context window actually is, because the buzzword makes it sound softer than it is. It's a fixed number of tokens the model can attend to on a single call — not a metaphorical "attention span," a hard byte-like budget you can count. Every system prompt, every tool result, every prior turn of the conversation competes for space inside that number, and once you're at the limit, admitting one more token means evicting one that's already there. There is no append-only mode. There's a budget, and you're always spending it.

The mistake underneath most bad context engineering is treating the window as memory instead of what it is: a cache. Memory is where you'd put something and trust it to still be there later because you decided it mattered once. A cache is where something earns its place continuously, gets evicted under pressure, and — crucially — where a miss has a cost that depends entirely on what got evicted. Model your context window as memory and your only verb is "append," which is fine for a six-turn conversation and catastrophic for a two-hundred-turn agent run, because the window doesn't grow to match — it fills, and then something you didn't choose starts falling out.

The other thing the window isn't: a place where more content is free upside. This is the part "context engineering" as a buzzword tends to skip, because "just add more context" sounds like the low-effort win. It isn't. The "Lost in the Middle" result (Liu et al., 2023) found that retrieval accuracy over a long input degrades measurably when the relevant fact sits in the middle of the context rather than at the start or end — so a bloated window doesn't just cost more tokens, it actively buries the fact you need under tokens that don't help, in the position where models are worst at finding it. Padding the window with "just in case" context is not a free hedge. It's a tax on the thing you actually needed to retrieve.

So: fixed budget, real eviction, and a penalty for stuffing it too full. That's the physical shape of the problem context engineering is supposed to solve. Everything else is a decision made inside that shape.

What gets kept, what gets dropped

Once you accept the window is a cache, the actual work is the packing decision, and there are two failure modes on either side of it, both of which I've watched break real runs.

The first is contamination: keeping the wrong thing. The default failure here isn't "too much junk in the window," it's a specific, sneaky case — a failed attempt sitting in the transcript as the most recent turn, quietly serving as a template for the next one. I've written about this at length: a naive retry that resends the full transcript including the model's own wrong answer doesn't give the model a clean second attempt, it gives the model a context window that says "here's a plausible-looking answer, do another one of those." The fix wasn't wiping the window — plenty of retries are transient failures where the original context was fine and a resend is correct and cheap. The fix was classifying why the attempt failed and, for the semantic failures, scrubbing the specific wrong turns while keeping a named statement of what's now ruled out. Keeping the literal failed reasoning around because "it might still be useful" is exactly the packing mistake: it's not useful, it's an anchor.

The second is compaction, which is the same problem from the opposite direction: dropping the wrong thing. Every long-running agent eventually has to summarize its own history to keep the window bounded, and that summarization is a lossy operation performed, by default, by a policy that doesn't know which facts are load-bearing. A recency-based compactor — keep the last N turns verbatim, summarize the rest — treats an early hard constraint ("never touch the production database," stated on turn 3) exactly like it treats a discarded exploration path from turn 40: both age out of the verbatim window and both get compressed with the same lossy summarizer, which means the constraint survives only by luck. I ran the numbers on this in that post: under a recency policy with a middling retention rate, a fact planted early and needed at the end survived to the final turn only 30% of the time across two hundred thousand simulated runs. Pin that same fact — exempt it from the generic summarizer entirely — and it survives 100% of the time, because pinning isn't a smarter guess about what to keep, it's a guarantee that removes the guess.

Notice these are the same mechanism pointed in opposite directions: contamination is a policy that's too generous with what it keeps, compaction is a policy that's too indiscriminate about what it drops. Both are cases of an eviction decision made without knowing the cost of a miss. The fix in both cases is the same shape: stop treating everything in the window as fungible text, and give the packing decision a schema — durable state that's pinned and auditable (constraints, IDs, the task goal, decisions already made) versus transient content that's genuinely fine to compress or drop (tool-call chatter, superseded attempts, resolved exploration). If you can't currently list what belongs in the durable bin for your own agent, that's not a detail to fill in later — that's the actual gap, and it's the same gap whether it shows up as a contaminated retry or a forgotten constraint.

What it costs to get wrong

Here's the part the "just write a better prompt" framing hides completely: bad context engineering isn't a quality problem you notice and shrug off, it's a cost problem that compounds, and it compounds quadratically if you let the window grow with the run instead of managing it.

The mechanism is simple once you see it. An agent step is a model call, and the model is stateless — every call re-sends the entire transcript so far as prefill. If you let that transcript grow by roughly a fixed amount every step and never evict anything, the prefill at step k is proportional to k, and the total tokens billed across an N-step run sum to the classic triangular number, N²/2. Doubling the run length doesn't double the bill. It roughly quadruples it — a 40-step run costs about four times a 20-step run, not two, and the ratio gets closer to a clean 4× the longer the run goes, because the fixed costs wash out and the quadratic term takes over. A short demo hides this entirely, because the term is small when N is small. It only bites once the run is long enough to matter, which is precisely when nobody's watching the per-step number anymore.

This is where context engineering and cost management are the same discipline wearing different names. The four moves that flatten that curve — truncating tool results to digests instead of carrying them whole, externalizing state to a scratchpad instead of the raw transcript, scoping sub-agents with their own fresh windows, and treating prefix caching as a discount rather than a fix — are exactly the "admit, evict, summarize" verbs the cache framing gives you. Compaction done deliberately and continuously, at every step, is what keeps the curve flat; compaction done in a panic at the context limit is what produces both the quadratic bill and the lost constraint, from the same root cause: nobody decided, in advance, what earns its place.

And the token line is often not even the expensive part of getting this wrong. A blown context budget doesn't just cost more per call — it produces the "Lost in the Middle" accuracy hit, which produces a wrong answer, which (if anything in your pipeline routes uncertain outputs to a human) turns into review time, and a human minute is one to two orders of magnitude more expensive than the tokens that triggered it. A packing mistake that looks like a rounding error on the token invoice can be the dominant line item on the real one, once you count what it costs downstream instead of just what it costs to prefill.

Context engineering as a discipline, not a one-time prompt

If the packing decision is made continuously and the cost of getting it wrong compounds, then "context engineering" only means something if it's a practice you can measure, not a document you write once. Three things make it a discipline instead of a vibe:

Make the durable bin explicit and testable. Don't trust that your summarizer "probably" keeps the important stuff — enumerate what must never be dropped (constraints, IDs, the task goal, decisions already made) and pin it structurally, outside the summarizer's reach. Then test it directly: can the agent reconstruct what it already ruled out, using only the compacted context? If not, you've found the gap before turn eighty finds it for you.

Measure the shape of your own window, not someone else's default. The right budget isn't the model's max context, it's a number you choose based on what the task needs, and the right length distribution for training or evaluation data is the one that matches what you actually serve — padding training examples to a tidy short length when production requests run long teaches a distribution you'll never see live, in exactly the same way that eyeballing "this feels like enough context" for an agent skips the measurement that would tell you whether it actually is.

Log every eviction and every scrub. The ruled-out list from a scrubbed retry and the pinned-fact audit from a compaction pass are both, underneath the terminology, the same artifact: a record of what your packing policy decided not to keep, and why. That log is what turns "the agent got worse on long runs" from a mystery you debug as a reasoning failure into a one-line check: was the fact it needed still in the window, yes or no. Most of what looks like an agent "forgetting" or "hallucinating" past turn fifty is not the model getting dumber. It's a packing decision nobody logged, made by a component that didn't know what it was throwing away.

None of this requires a bigger context window, a better model, or a cleverer prompt. It requires treating the input to every call as a resource with a budget, an eviction policy, and a cost function — which is what "context engineering" should have meant from the start, before the term got used to rebrand the thing everyone was already doing on autopilot.

What I'd actually do

1. Stop calling it a prompt and start calling it a packing decision. The frame change alone surfaces the actual question: what's in this call's window, and who decided it should be?
2. Enumerate the durable bin before you write a summarizer. If you can't list the handful of facts that must survive to the end of a run, you don't have a compaction policy — you have a hope.
3. Never let a scrub or a compaction pass go unlogged. The record of what you dropped is the only thing that turns a downstream failure into a fast diagnosis instead of a re-read of the whole transcript.
4. Budget the window like you budget the token bill — because it's the same budget. A curve that grows linearly with run length is a design choice, not a law of nature; the quadratic version is what happens by default when nobody makes the choice.
5. Test your packing policy the way you'd test any other production system. Can the agent answer, from the compacted context alone, "what have you already ruled out, and what were you told never to do?" If it can't, ship the fix before you ship the feature.

The window is bounded. Something is always getting evicted, whether or not you're the one deciding what. Context engineering is just the name for making that decision on purpose instead of leaving it to whichever summarizer or retry loop happens to run first.