Originally published on The AI Prism
The prompt is no longer the unit of work
For the first three years of practical LLM application building, the craft everyone discussed was prompt engineering: the art of finding the exact words, examples, and formatting tricks that coaxed a model into the right answer. The skill centered on a single message. Anthropic now frames that era as concluded. In a September 2025 post, the company described context engineering as “the natural progression of prompt engineering,” shifting the central question from “what words do I write?” to “what configuration of context is most likely to generate the model’s desired behavior?” [Anthropic, Effective context engineering for AI agents].
The distinction matters because modern systems rarely answer one query and stop. An agent running in a loop generates new data at every step: tool outputs, retrieved documents, intermediate reasoning, and user corrections. As Anthropic puts it, context engineering is “the art and science of curating what will go into the limited context window from that constantly evolving universe of possible information.” The prompt is only one tributary feeding a much larger river, and the discipline’s center of gravity has moved upstream to the whole pipeline that assembles the window.
This reframing is not merely semantic. It changes who does the work and when. Prompt engineering is a discrete authoring act performed before a request. Context engineering is a continuous systems concern performed by code, by the model, and by retrieval and memory subsystems at every turn. Treating them as the same job understates how much the build has changed. It also changes how success is measured: prompt quality was judged by one-shot output, while context quality is judged by the agent’s behavior across an entire long-running session, where early mistakes compound.
What context engineering actually means
Context is the full set of tokens the model samples from at any moment, and the engineering problem is optimizing the utility of those tokens against hard constraints. Context engineering is the discipline of curating and maintaining that set across the life of a task, including “all the other information that may land there outside of the prompts” [Anthropic]. The components are mundane but consequential: system instructions, tool definitions, Model Context Protocol (MCP) servers, retrieved external data, message history, and persistent memory.
Unlike writing a prompt, which is a one-time act, context engineering is iterative and continuous. The curation step happens every time the system decides what to pass to the model. The practitioner is no longer a wordsmith polishing a sentence; they are a systems designer managing a data pipeline that must stay high-signal as it grows and as the task forks into unforeseen directions. Neo4j makes the contrast explicit: prompt engineering “treats context as static,” while agents “can only behave reliably when their context keeps pace with the decisions they make and the data they uncover” [Neo4j].
A useful mental model is that context engineering sits one layer below prompting. The prompt is a request; the context is the entire environment in which that request is interpreted. Improving the request while ignoring the environment produces diminishing returns once the environment is noisy, stale, or bloated. This is why teams that shipped a strong prompt in 2023 and then scaled to agents in 2025 often report regressions: the single-turn craft does not transfer, because the failure modes move from wording to information flow.
Context is a finite, decaying resource
The single most important fact about context is that it degrades with size. Research on needle-in-a-haystack benchmarks exposed a phenomenon called context rot: as the number of tokens in the window grows, the model’s ability to recall information from that context declines [Chroma, Context Rot]. This is not a quirk of one model. Anthropic notes it “emerges across all models,” producing a performance gradient rather than a hard cliff: models stay capable at longer contexts but lose precision on retrieval and long-range reasoning.
The root cause is architectural. Transformers let every token attend to every other token, which creates n-squared pairwise relationships across n tokens. As the window stretches, the model’s “attention budget” gets spread thin, and its training distribution, which skews toward shorter sequences, leaves it less equipped for very long-range dependencies. Position encoding interpolation helps models handle longer sequences, but with some degradation in position understanding.
The practical conclusion is blunt and counterintuitive for anyone trained to “give the model more context”: a smaller, tightly curated context usually outperforms a large, padded one. Treating context as a precious, finite resource with diminishing marginal returns is the foundation of the entire discipline, and it is why subtraction, not addition, is the recurring theme.
The Claude 5 moment: deleting 80% of a system prompt
The clearest evidence that the rules changed came from Anthropic itself. In July 2026, the company published “The new rules of context engineering for Claude 5 generation models,” reporting that it removed over 80% of Claude Code’s system prompt for models like Opus 5 and Fable 5 “with no measurable loss on our coding evaluations” [Anthropic, New rules of context engineering]. The post reached 463 points on Hacker News, a signal of how much the shift resonated with practitioners living inside these tools daily.
Anthropic’s diagnosis was that it had been overconstraining the model. Old transcripts showed conflicting messages stacked in a single request: “leave documentation as appropriate” sitting next to “DO NOT add comments.” Those guardrails were once necessary to prevent worst-case behavior in weaker models, but newer models can use surrounding context and judgment instead. The reporting is a useful data point because it comes from the team with the most context-engineering surface area in production, not from a vendor selling a framework.
The episode also reframes what “good prompting” means for advanced models. Where earlier advice optimized phrasing, the Claude 5 lesson optimizes for restraint: remove what the model no longer needs, and let capability substitute for instruction. That is a different skill set, closer to editing than to writing. For teams maintaining their own agents, the takeaway is to schedule regular context audits, because the optimal system prompt for a weaker model becomes dead weight and conflicting noise for a stronger one released a few months later.
From rules to judgment: the six expired myths
The Claude 5 post catalogs specific best practices that became myths as models improved. Each pair is a small window into how context engineering evolves with capability:
Give rules → use judgment. Instead of “default to writing no comments,” the new prompt says “write code that reads like the surrounding code.” Give examples → design interfaces. Anthropic found that giving tool examples “actually constrains them to a certain exploration space,” so it now focuses on expressive tool parameters instead. Put it all upfront → progressive disclosure. Verification and code review moved into separate skills the agent calls only when needed [Anthropic].
The remaining shifts reinforce the pattern: repeat yourself → simple tool descriptions, moving instructions into tool definitions rather than the system prompt; memory in CLAUDE.md → auto-memory, where Claude now saves relevant memories without manual hotkeys; and simple specs → rich references such as HTML artifacts, test suites, or rubrics that spin up verifier agents. The throughline is subtraction. Better models let you delete scaffolding and trust the system to assemble context at the right time, which lowers maintenance cost as a side benefit.
Memory and retrieval: external notebooks and just-in-time context
Once a task spans many turns, the context window alone cannot hold everything. Structured note-taking, or agentic memory, is the practice of having the agent write notes to a store outside the window. Anthropic cites its own Claude playing Pokémon experiment: without any memory-structure prompting, the agent developed maps of explored regions, tracked unlocked achievements, and maintained combat notes across thousands of steps [Anthropic]. After context resets, it reads its own notes and continues multi-hour sequences that would be impossible if everything lived in the window.
The design implication is that memory is not a passive dump; it is a curated, queryable system. Weaviate’s framework separates memory from retrieval, warning that “old, low-quality, or noisy entries eventually come back through retrieval and start to contaminate the context” [Weaviate, Context Engineering]. Periodic pruning, merging duplicates, and replacing long transcripts with compact summaries keep retrieval sharp. The retrieval half of the system deserves equal attention, because a memory store full of stale facts is worse than no memory at all. Governance matters here: without a policy for what gets written, how long it persists, and who can read it, memory becomes a source of silent drift rather than a reliable record of past decisions.
The newer retrieval pattern is just-in-time (JIT) context: the system holds lightweight identifiers such as file paths, saved queries, and links, and loads actual data only when the agent decides it needs it. Anthropic describes this as mirroring human cognition, where we rely on file systems and bookmarks rather than memorizing entire corpuses [Anthropic]. Claude Code is the reference implementation, dropping CLAUDE.md in up front while using glob and grep to fetch files on demand. The trade-off is speed: runtime exploration is slower than precomputed retrieval and demands thoughtful tool design, or the agent wastes context chasing dead ends.
Compaction and long-horizon coherence
For tasks that genuinely exceed the window, compaction is the first lever: summarize a conversation nearing its limit, then reinitialize a fresh window carrying the summary plus the few most recently accessed files. The art is in what to keep. Anthropic warns that “overly aggressive compaction can result in the loss of subtle but critical context whose importance only becomes apparent later.” Its recipe is explicit: “start by maximizing recall to ensure your compaction prompt captures every relevant piece of information from the trace, then iterate to improve precision” [Anthropic].
Compaction is one of three coherence techniques, alongside note-taking, which excels for iterative work with clear milestones, and sub-agent architectures, which handle parallel research. In a sub-agent design, specialized agents explore with tens of thousands of tokens but return only a 1,000-2,000 token distilled summary, keeping the lead agent’s window clean. This separation of concerns showed a substantial improvement over single-agent systems on complex research tasks. The choice depends on the task’s shape, but all three exist to solve the same problem: preserving signal across time without overflowing the window.
Engineers implementing compaction should tune the prompt on real agent traces rather than guessing. Recall-first, precision-second is the safe ordering because a missed fact is rarely recoverable, while superfluous content can be trimmed in later iterations without permanent loss. It is worth measuring the cost of compaction directly: a poorly tuned summarizer silently degrades the agent’s memory of earlier constraints, and the failure shows up as the agent contradicting decisions it made an hour of tokens earlier. Logging before-and-after traces is the only reliable way to catch that. The technique is forgiving of excess but unforgiving of omission, which is why the recall-first heuristic dominates in practice.
Tool design is context design
A frequently missed insight is that tools are context. Every tool definition consumes tokens and shapes the agent’s decision space. Anthropic’s guidance is to keep tools self-contained, robust to error, and unambiguous in purpose. A common failure mode is a bloated tool set with overlapping functionality; if a human engineer cannot say which tool fits a situation, the agent will not do better. Curating a minimal viable set of tools also makes long-horizon context pruning easier, because fewer definitions compete for attention over a long session.
The Claude 5 post pushes this further: rather than feeding examples of tool use, design the interface so the parameters themselves teach the model. A status field exposed as an enumeration between pending, in_progress, and completed both hints at usage and sets the expected behavior. Good tool design reduces the need for explicit instructions elsewhere in the context, which is exactly the kind of subtraction the discipline rewards. Token-efficient tool outputs matter as much as clear inputs, because bloated returns refill the window just as fast as verbose prompts.
Tool design also interacts with retrieval. Deferred-loading tools, which the agent must look up before using, let a system carry many capabilities without paying their context cost until they are actually invoked. This is progressive disclosure applied at the tool layer, and it is one of the cheaper wins available to teams building their own harnesses. The same principle applies to skills and reference files: keep them discoverable but unloaded, and let the agent pull them in only when the task demands. The architecture that scales is one where the baseline context is small and almost everything expensive is fetched on demand.
What practitioners should actually learn
The shift to context engineering changes the daily work of anyone building agents. First, stop over-specifying. If a newer model can infer intent from surrounding context, delete the rule. Anthropic shipped a claude doctor command to right-size skills and CLAUDE.md files automatically, a sign that cleanup is now a first-class maintenance task rather than an afterthought. Second, invest in retrieval and memory hygiene rather than longer prompts; a small set of high-signal tokens beats a large padded window every time.
Third, treat just-in-time disclosure as the default architecture, not an optimization. Build trees of files and skills that load on demand instead of a monolithic instruction block. Fourth, design tools as interfaces, not as things to be exemplified. These lessons apply well beyond coding agents; they scale to any multi-turn system, including the education and tutoring agents surveyed in Andrew Ng’s plan to rebuild education with AI, and to the security-sensitive agent deployments examined in the AI worm already crawling through Copilot for Word.
The deeper question is whether context engineering is a stable destination or just the current name for an endless moving target. As models grow more capable, they need less prescriptive scaffolding, which suggests the craft will keep shrinking toward curation and away from construction. The risk for teams is investing heavily in hand-tuned context pipelines that a model generation later renders unnecessary, the same way hand-coded prompts aged out. If a future model maintains its own memory, retrieval, and compaction internally, what exactly is left for the engineer to engineer?
References
• Anthropic. “Effective context engineering for AI agents.” Sep 29, 2025. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
• Anthropic. “The new rules of context engineering for Claude 5 generation models.” Jul 24, 2026. https://claude.com/blog/the-new-rules-of-context-engineering-for-claude-5-generation-models
• Chroma. “Context Rot: Why your long context LLM fails.” https://research.trychroma.com/context-rot
• Weaviate. “Context Engineering – LLM Memory and Retrieval for AI Agents.” https://weaviate.io/blog/context-engineering
• Neo4j. “Why AI teams are moving from prompt engineering to context engineering.” https://neo4j.com/blog/agentic-ai/context-engineering-vs-prompt-engineering/
If a future model maintains its own memory, retrieval, and compaction internally, what exactly is left for the engineer to engineer?
The post Context Engineering Replaced Prompt Engineering appeared first on The AI Prism.
Cross-posted from theaiprism.com — Cutting Through the AI Noise 🧊
Top comments (0)