Context engineering for people who already know how to prompt

#ai #machinelearning #llm #programming

If you've spent the last two years getting good at prompting, you've probably been told — loudly, repeatedly, all through 2026 — that the skill is now dead and "context engineering" has replaced it.

Ignore that framing. It's wrong, and believing it will make you worse at both.

Here's the accurate version: prompt engineering is a subset of context engineering. Prompting is what you say to the model. Context engineering is the discipline of curating everything the model can see when you say it — the system prompt, tool definitions, retrieved documents, prior turns, and memory. Your carefully written prompt is one (load-bearing) component inside that larger surface. Anthropic, who did the most to popularize the term, still lists "system prompt design" and "few-shot prompting" as context-engineering techniques in their own write-up on the topic. Nothing died. The surface just got bigger.

This post is for engineers who already prompt well and want the next layer: what context engineering actually is mechanically, and the handful of patterns worth internalizing.

Why a bigger context window didn't save you

The seductive idea of 2025 was that long context windows would make all of this irrelevant — just stuff everything in and let the model sort it out. It didn't work, and it's worth understanding why, because the why is the whole basis for the discipline.

A model with a million-token window does not have a million tokens of usable attention. Transformer attention scales with the square of the sequence length — every token has to relate to every other token — so attention is a finite budget that every token you add draws down. More input doesn't just cost money and latency; it dilutes the model's ability to focus.

This shows up empirically in two ways you should know by name:

Lost in the middle. Models reliably use information best when it sits at the start or end of the context, and measurably worse when the relevant content is buried in the middle — even for models explicitly marketed as long-context. This is the Liu et al. paper that named the effect, and a follow-up showed the U-shaped attention bias is intrinsic: beginning and end tokens get more attention regardless of relevance.
Context rot. Chroma ran the most rigorous test I've seen — 18 models, including the latest GPT, Claude, Gemini, and Qwen releases — and found recall and reasoning degrade as input length grows, even on deliberately trivial tasks. The finding that should bother you: models often scored better on shuffled context than on logically coherent documents. Capacity is not utilization.

So the goal of context engineering, in Anthropic's words, is "finding the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome." Not the biggest context. The smallest one that works.

The patterns worth internalizing

Once you accept that the window is a budget, the practices fall out of it. Here are the six I'd make sure you actually understand — each with what to do and the failure mode it prevents.

Pattern	What to do	Failure it prevents
Attention budgeting	Treat tokens like a budget. Spend on high-signal content; ruthlessly cut boilerplate, dead history, and redundant tool docs.	Context rot — quiet degradation as the window fills.
Retrieval placement	Put the most relevant retrieved chunks at the start or end of the window, not the middle. Prefer just-in-time retrieval (load via tools at runtime) over front-loading everything.	Retrieving the right document and still getting a wrong answer because it landed mid-context.
Tool-result formatting	Return condensed, high-signal tool output. For big payloads, return identifiers/handles the agent can expand on demand instead of dumping full objects.	One fat API or SQL response blowing the budget and evicting relevant context.
Structured output / schema design	Use native structured outputs / constrained decoding; keep schema nesting to 2–3 levels.	Malformed JSON breaking your pipeline; deep nesting raising error rates.
System vs. turn separation	Keep durable rules and persona in the system prompt at the right altitude — specific enough to guide, loose enough not to be brittle. Keep per-turn data out of it.	Brittle over-specification, prompt drift, and broken prompt caching.
Memory & compaction	Persist state outside the window (progress notes, files) and re-hydrate selectively; when nearing the limit, summarize and reinitialize with the distilled state.	Losing the task thread the moment the work outgrows a single window.

Retrieval is a deep topic in its own right — if you're going further there, the questions teams ask about RAG in interviews map the common failure modes pretty well — but for context engineering the load-bearing insight is placement and restraint, not just "did we retrieve."

The memory pattern is the one most people underrate. Anthropic's long-running agent work leans heavily on it: agents that write structured progress files and read them back at session start stay coherent across hundreds or thousands of steps that could never fit in one window. The window is working memory; durable state lives outside it.

Misconceptions worth killing

"Prompting is dead." No — it's a subset. The system prompt and your few-shot examples are first-class context-engineering components. You cannot out-engineer a badly written instruction. If you want to pressure-test what "good prompting" even means, the fundamentals people drill for prompt-engineering interviews are a fair checklist — that layer matters more now, not less.
"Just use a bigger window." The Chroma study refutes this directly: performance degrades with length even on easy tasks, and coherent long documents did worse than shuffled ones. Capacity ≠ utilization.
"More context is better." More context dilutes the attention budget and raises the odds your key fact lands in the dead zone in the middle. Smallest high-signal set wins.
"Context engineering is just RAG." RAG is one mechanism. Tool-result formatting, memory, compaction, schema design, and system/turn separation are all in scope. RAG without placement and budgeting discipline still fails.

The numbers I'm deliberately not citing

If you read around on this topic you'll keep hitting two figures: an "82% of leaders say prompting alone isn't enough" stat, and a "task completion jumped from 83% to 96% with context engineering" claim. I tried to source both. The 82% traces back to a vendor survey with no published methodology from a company that sells context tooling. The 83%→96% appears to be two unrelated benchmarks stitched together into a before/after that nobody actually ran.

So I'm not repeating either, and neither should you. The mechanism above — finite attention, position bias, measurable context rot — is real and reproducible. You don't need the fake precision to justify the practice.

Where this leaves you

Context engineering isn't a new thing that killed an old thing. It's the name for the wider surface that good prompting always lived inside — we just couldn't ignore the rest of the window anymore once agents and tools and retrieval entered the picture. It's quietly become part of the baseline for AI engineering roles: it shows up in interviews now, not just in production post-mortems.

If you already prompt well, you're not starting over. You're learning to manage the budget your prompt was always spending.

Further reading: