TL;DR — Most context assembly code is still string concatenation with truncate-when-full heuristics dressed up as 'context engineering.' Treating the context window as a scarce, priced resource — with priority tiers, eviction policies, and testable invariants — is the difference between an engineering discipline and prompt folklore. This piece argues for building a real scheduler for context, not another templating layer.
Every team building on LLMs eventually writes the same piece of code: a function that takes a system prompt, some retrieved chunks, a chat history, a few tool schemas, and maybe some examples, and glues them into a single string before it hits the token limit. Then it truncates from the oldest message, or drops the lowest-ranked chunk, or just prays. That function gets called "context engineering." It is not engineering. It is string concatenation with a safety valve.
The industry renamed prompt engineering to context engineering because everyone realized the prompt was never the interesting part — the assembly of everything around the prompt was. But the tooling didn't follow the rename. We still treat the context window like a document to be formatted instead of what it actually is: a fixed, expensive, contested resource that multiple consumers are competing for on every single request.
The template engine is the wrong abstraction
Jinja, f-strings, and their cousins are the dominant tool for context assembly, and they are fundamentally the wrong shape for the problem. A template engine answers the question "how do I format this data into text." It has no opinion about what happens when the formatted text doesn't fit. It has no concept of priority. It has no notion that a retrieved passage and a system instruction are not interchangeable just because they're both strings.
Compare this to how every other resource-constrained system in computing behaves. An OS memory allocator doesn't format your data into RAM and then panic if it overflows — it has an eviction policy. A query planner doesn't concatenate every possible join and hope the database figures it out — it costs each operation and picks a plan under a budget. Context assembly is the one place in the stack where we skipped straight from "no resource management" to "ship it" without ever building the allocator.
A context window is a resource, not a document
The practical consequence of the document mental model is that token budget decisions get made in the wrong place, at the wrong time, by the wrong logic. Truncating the oldest chat turn is a decision about recency. Dropping the lowest-similarity chunk is a decision about retrieval score. Neither of these is a decision about value to the current task, which is the only thing that should actually determine what survives when the budget is tight.
A system prompt containing your safety instructions and a retrieved paragraph about a tangential topic are not the same kind of content, but a truncation-by-position or truncation-by-score policy treats them identically once they're both just tokens in a buffer. This is how you get the failure mode every team has seen and nobody logs: the model quietly stops following an instruction that got pushed out of the window three turns ago, and the first anyone notices is a support ticket, not a stack trace. There is no exception thrown when context degrades. That silence is exactly why this needs to be engineered deliberately instead of discovered in production.
What an actual scheduler needs
If you accept the resource framing, the shape of the fix follows from decades of prior art in operating systems and query planning. Three things are non-negotiable.
Priority tiers, not a flat list. Every piece of content entering the context should be tagged with a tier: pinned (system instructions, safety constraints, current user turn — never evicted), structural (tool schemas, output format — evicted only as a last resort), and evictable (retrieved passages, few-shot examples, older history — evicted first, and evicted in a defined order within the tier). This alone eliminates the most common silent failure: instructions getting bumped by data.
An explicit eviction policy per tier, not a global truncation rule. "Drop the oldest message" is fine for chat history and catastrophic for retrieved evidence, where the most relevant chunk might be the one you fetched five turns ago. Each tier needs its own rule — recency for history, relevance score for retrieval, frequency-of-use for examples — because they fail differently and should be evicted differently.
A cost-per-token value estimate, not a boolean include/exclude. The real question a scheduler answers isn't "does this fit," it's "is this worth the tokens relative to everything else competing for the same budget." A tool schema that's used in one out of every fifty calls is expensive in a way a two-line instruction is not, even if they're the same token count. Treating all tokens as fungible is the same mistake as treating all memory pages as equally hot.
The missing artifact: a diffable context
Here's the part that actually makes this an engineering discipline rather than a design philosophy: none of the above is useful unless you can inspect and diff it. Right now, when a model's behavior shifts after a prompt template change, a retrieval index update, or a new tool being added, most teams debug it by re-running the request and eyeballing the output. That's the equivalent of debugging a build failure by staring at the binary.
What's missing is a build artifact: the fully assembled context, with every piece tagged by source, tier, and token cost, emitted alongside the request as a structured object rather than a flattened string. With that artifact, "why did the model ignore my instruction" becomes a diffable question — you can literally show that the instruction was present in build A and evicted in build B, and see exactly which competing content pushed it out and why the policy chose that eviction. Without it, you're debugging by vibes, and vibes don't scale past the second person on the team.
Testing this like a scheduler, not a snapshot
Snapshot testing prompts — asserting the assembled string matches a golden file — breaks on every trivial content change and tells you nothing about whether the policy is correct. The tests that actually matter are invariant tests: pinned content is never evicted, under no combination of inputs does the assembled context exceed budget, higher-priority evictable content is never dropped while lower-priority content survives, and total assembled tokens plus generation budget never exceeds the model's actual context limit with margin for the tokenizer's own overhead.
These are the same kinds of properties you'd write for a real scheduler, because that's what this is. The teams currently debugging degraded agent behavior by re-reading transcripts are doing the manual-QA equivalent of testing a memory allocator by running the program and hoping it doesn't crash. It's not that context engineering lacks rigor because the problem is unrigorous. It's that we built a formatting tool for a resource-allocation problem, and formatting tools don't have policies, invariants, or diffs — because they were never designed to need them.
Top comments (0)