Ask ten AI teams what they do and most will say context engineering. Two years ago they said prompt engineering. The label changed fast, faster than infrastructure usually does.
Open the codebase behind that label and you'll typically find the same thing you'd have found in 2024: a system prompt, a retrieval call bolted on with string concatenation, and a truncation function that fires when the token count gets uncomfortable. The vocabulary evolved. The system underneath it mostly didn't.
That gap is the real story. It belongs to systems and infrastructure teams, not to whoever happens to write the prompts.
The Term Won. The System Didn't.
Here's the imposter version, the one half a dozen job postings currently list as a core skill: fetch a few chunks from a vector store, drop them above the user's question, ship it. That's retrieval-augmented prompting. It's useful, and it's also not a system, because nothing about it persists, nothing about it degrades gracefully, and nobody owns it once it's deployed.
The real version has three separate jobs running underneath one label, and they rarely live in the same part of the stack. State has to be stored somewhere between requests. Context has to be assembled from multiple sources within a latency budget, on every single call. And whatever gets loaded has to be pruned on a schedule, before stale content quietly degrades the model's output. Most teams that claim context engineering have built exactly zero of these three.
State Has to Live Somewhere
Persistent context state is a storage decision, not a prompt decision, and most teams haven't made it on purpose. They've defaulted into statelessness, then treated the resulting amnesia as a model limitation instead of an architecture choice nobody made deliberately.
Anthropic's memory tool, shipped for the Claude Developer Platform, takes a specific position here. Claude reads and writes files in a memory directory that the developer hosts. The model only issues the read and write instructions. Where those files actually live, local disk, object storage, a database, is entirely the developer's call.
That's a real tradeoff, not a solved problem. File-based storage is simple and auditable, but it doesn't give a team the query flexibility a vector store or relational schema would if they need to search across sessions instead of just replaying one. It also pushes versioning and conflict handling up into the application layer instead of a database engine. Teams that skip this decision by default end up with context that resets every session, then wonder why the agent forgets debugging context from three days earlier.
Assembly Is a Pipeline, Not a Copy-Paste
Getting the right context in front of the model, on every call, is an assembly step with its own latency budget and its own failure modes. It deserves to be treated like any other production service, not like glue code written once in a notebook.
This is what protocols like Model Context Protocol (MCP) are actually solving. MCP standardizes how a model requests data and tool access from external sources, instead of every team hand-rolling its own integration between the model and whatever systems hold the real information. The comparison worth making is an API gateway: one slow backend shouldn't take the whole request down with it, and one slow context source shouldn't either.
Most teams haven't built that resilience. Three upstream calls, a vector store, a ticketing system, an internal wiki, get made synchronously in the request path with no cache and no fallback. It works in the demo, because the demo never hits the slow dependency. It stops working the first time one does, and there's no owner for the timeout that should have caught it.
Pruning Needs to Be a Service, Not a Cron Job Someone Forgot
Deciding what gets cleared from context, and when, has to run continuously in production. It cannot be a decision made once in a design doc and left alone.
Logging went through the same evolution years ago. Print statements were fine for a single developer debugging locally. Production systems needed structured logging, retention policies, and alerting, because the volume and the stakes changed. Context pruning is sitting at the print-statement stage in most organizations right now.
Anthropic's own documentation makes the direction explicit. Context editing clears stale tool results once a conversation crosses a defined threshold. That capability was originally something developers configured client-side through an SDK. Client-side compaction is now deprecated in favor of server-side compaction, which Anthropic recommends because it handles token accounting and summarization automatically instead of asking every developer to rebuild that logic themselves. A truncation function someone wrote eight months ago doesn't know about a new content type or a changed usage pattern. It just quietly breaks.
Nobody Owns This Yet
None of these three jobs has a natural home in most org charts. The AI or ML team treats context as a prompting problem, because that's the layer they can see and measure. Platform engineering doesn't know it's in scope, because nobody told them a context pipeline needs the same on-call rigor as a payments API. Whoever owns data usually owns storage for everything except the one system actually feeding the model in production.
That gap doesn't show up in a demo. It shows up on turn forty of a long-running agent session, when the model references something that should have been pruned three exchanges ago, or forgets something it stored two days earlier because nobody built the retrieval path for old memory files. When it happens, the on-call engineer gets paged with "the agent is being weird," not a clear signal like "assembly exceeded latency budget," because nobody built a dashboard for a system nobody agreed was a system.
Winning the naming war was the easy part. A term spreads because it's useful shorthand, and context engineering is genuinely better shorthand than prompt engineering for what these systems now do. But shorthand raises expectations. Say the phrase out loud in 2026 and the person listening assumes a state store, an assembly pipeline, and a pruning policy exist behind it.
Most of the time, they don't.
It's a liability, sitting quietly in production, waiting for turn forty.
Top comments (0)