For the last few months, I have been building my own AI framework for software development. It contains instructions for different engineering activities: implementing features, refactoring, writing tests, debugging, working with APIs, reviewing code, and many smaller tasks. As the framework grew beyond 100,000 characters of instructions, one problem became increasingly important: how do I give an AI agent exactly the instructions it needs without loading the entire framework into its context?
While working on this problem, I ended up testing two different architectures: Agent-Driven Context Discovery and a Prompt Engine with Context Compilation. Both allow you to keep instructions modular and load only what is relevant to the current task. The major difference is who is responsible for assembling that context.
Agent-Driven Context Discovery leaves this responsibility to the agent. You give it an entry point, usually a base instruction file, and that file references other instructions. The agent reads those files, follows their references, and gradually discovers the context required for the task.
Prompt Engine with Context Compilation moves most of this work into deterministic code. The agent still selects the type of activity it is about to perform, but once that decision is made, the Prompt Engine deterministically resolves the required instructions and compiles them into a ready-to-use context. The agent no longer has to manually discover and retrieve every instruction.
Shared Foundation: Intent Router / Playbook
Both architectures still need a mechanism that connects a task with the appropriate instructions. I call mine a playbook.
The playbook is essentially an intent router. It describes the types of activities the framework supports and maps those activities to the instruction sets they require. Implementing functionality, refactoring code, writing tests, debugging, and reviewing code are examples of activities that can have different instruction dependencies.
With Context Discovery, this routing can live inside the instruction graph itself. The base instruction tells the agent where to go, and the agent follows the appropriate path.
With my Prompt Engine, I use an MCP tool called getInstructions. It accepts typed activities and uses them to programmatically determine which instructions should be compiled. The agent therefore makes one high-level decision — what am I about to do? — while deterministic code handles the low-level question — which exact instructions does that activity require?
Approach 1: Agent-Driven Context Discovery
This was the first architecture I implemented. I split my framework into many small instruction files and created references between them. The agent received a base instruction as its entry point and was expected to follow those references whenever additional instructions were required.
Conceptually, it looked something like this:
Task → Base Instructions → Agent → Referenced Instructions → More Instructions → Work
I liked this approach initially because it was extremely easy to build. The instruction system remained modular, individual files could be reused, and adding a new rule usually meant adding or updating a reference. There was no separate infrastructure responsible for assembling prompts because the agent itself effectively acted as the resolver.
After using this architecture for some time, however, I started noticing several problems.
Problem 1: Reasoning Overhead
Every time the agent encounters a reference, it has to understand why that instruction exists, decide whether it should follow it, retrieve the file, process it, and potentially repeat the process. These are small decisions, but they are still decisions made by the model instead of deterministic operations performed by software.
I increasingly disliked spending the agent's reasoning capacity on this kind of plumbing. I want the model thinking about architecture, requirements, edge cases, and implementation decisions. Resolving an instruction dependency does not require intelligence if I can already express that dependency in code.
Problem 2: Reliability
I repeatedly saw agents skip files even when another instruction explicitly told them to read those files. It did not happen every time, but it happened often enough that I could not consider instruction delivery deterministic.
This creates an unpleasant failure mode. Your agent may produce perfectly reasonable code while silently missing one of the rules it was supposed to follow. The problem is not that the model misunderstood an instruction — it never loaded that instruction in the first place.
Problem 3: Tool Usage
Reading each instruction file requires a tool call. With more than 100,000 characters distributed across many files, the agent could make a surprising number of calls just to prepare itself for the actual task.
I repeatedly hit Copilot’s maximum tool-call limit while the agent was navigating instruction files. When that happened, Copilot stopped before the actual task was completed. That was a strong signal that I needed a different architecture.
Problem 4: Inefficient Context Usage
Another problem is that routing instructions remain in the agent's context long after they have served their purpose. Instructions such as "read this file," "for this activity, also read that file," or "if this condition applies, load these additional instructions" are useful during context discovery, but provide little value once the agent starts working on the actual task. From that point on, they simply consume space in the context window.
This problem also has the potential to grow over time. If the agent occasionally fails to load the right files for certain activities, the natural response is to make the routing instructions more explicit: add explanations, introduce more specific terminology, describe additional conditions, or repeat important references. That may improve routing reliability, but it also increases the amount of temporary information that remains in the context throughout the task.
In other words, the more effort you put into making agent-driven context discovery reliable, the more context you may end up spending on instructions that become useless immediately after discovery is complete.
Approach 2: Prompt Engine with Context Compilation
My second implementation moved instruction resolution out of the agent and into code.
The framework still contains dozens of small, reusable instruction files. The difference is that the agent no longer needs to navigate them one by one. A script resolves the dependencies, collects the required files, and returns the complete instruction set in a single operation.
The architecture now looks closer to this:
Task → Activity → Prompt Engine → Compiled Context → Agent → Work
The change may sound small, but the practical improvement was substantial. Once the agent selects an activity, the rest of the process is deterministic. If activity X requires instructions A, B, C, and D, the Prompt Engine delivers all four. There is no intermediate step where the agent can forget to open C or decide that D is unnecessary.
Context delivery also became much faster. Instead of making a sequence of tool calls, reading files one by one, and reasoning about what to retrieve next, the agent makes a single request. The Prompt Engine resolves the instruction graph programmatically and returns the compiled context in a single operation. This removes both the latency of repeated tool calls and the reasoning overhead involved in navigating the instruction structure.
Another important improvement was eliminating a practical limitation I repeatedly hit with Copilot: the maximum number of tool calls available to the agent. With Context Discovery, the agent regularly exhausted this limit while navigating and reading instruction files, which caused Copilot to stop before the task was completed. After moving instruction resolution into the Prompt Engine, instruction delivery requires only one tool call, so this failure mode effectively disappeared from my workflow.
In practice, I got three immediate benefits: the required instructions started reaching the context reliably, context assembly became much faster, and instruction discovery stopped consuming the agent's limited tool calls. Combined, these changes produced a very noticeable improvement in the overall speed and reliability of my workflow.
Problem 1: Your Prompt Engine Becomes a Mini-Product
The first downside is additional infrastructure. In my case, the Prompt Engine requires a custom MCP server that exposes the tools the agent uses to request its instructions. Once you introduce this layer, you effectively have another small software product living inside your repository.
Like any other software, it needs maintenance. You need to keep the implementation clean, handle edge cases, and ideally cover its critical behavior with automated tests. As the instruction system evolves, the Prompt Engine has to evolve with it.
This is simply the price of moving from a lightweight instruction structure to a more sophisticated architecture. You gain more deterministic and reliable context delivery, but you also introduce another component that you are responsible for maintaining.
Problem 2: You Can Accidentally Create Two Sources of Truth
Another challenge is instruction representation. The format that is convenient for programmatic processing is not necessarily the format that is convenient for a human. For example, the Prompt Engine may work better with structured JSON, while I would much rather read, write, review, and maintain the same instructions as Markdown.
The obvious solution is to keep both representations, but that immediately creates another problem: they can drift apart. If JSON says one thing and Markdown says another, you effectively have two sources of truth and can no longer be certain which representation describes the actual behavior of the system.
A better architecture needs one canonical source and a deterministic way to generate the other representation from it. Ideally, the two formats should also remain structurally similar enough that you can debug the agent-facing JSON while reading the human-friendly Markdown. Otherwise, the convenience of having two representations can quickly turn into another maintenance problem.
Problem 3: Compiled Instructions Can Pollute the Context
A Prompt Engine can also make context usage worse if the routing rules are not strict enough. Imagine that the current session already contains the instructions for API engineering, but the agent does not clearly understand that those instructions should not be requested again. It may call the instruction tool a second time and inject another full copy of the same compiled context into the conversation.
If your instruction sets are large, this can become surprisingly expensive. Instead of optimizing context usage, you may end up filling the context window with repeated copies of the same material. The more often the agent re-requests instructions, the faster this duplication compounds.
My recommendation is to add a hard guardrail at the infrastructure level.
Track which instruction sets have already been delivered during the current session, and if the agent requests the same one again, do not return the full instruction set.
Return a short message saying that the instruction is already present in the current context and should be reused.
I also make this failure mode explicit. If the agent believes the required instruction is actually missing despite the guardrail, it should treat that as a critical inconsistency, stop, and report the problem to the user rather than silently requesting or reconstructing the same context again.
This is one of those cases where a small deterministic safeguard can prevent a large amount of unnecessary context duplication.
Note: There Is Still One Non-Deterministic Step
This architecture does not make everything deterministic. The agent still needs to select the correct activity when it calls getInstructions, and that selection is an LLM decision.
For this reason, I think the tool interface should be strongly typed and the playbook should be designed carefully. Activity names and descriptions need to be clear enough that the model can reliably map real tasks to them. Otherwise, you simply move the failure point from instruction discovery to activity classification.
I would also recommend watching this routing closely when introducing such a system. Check which activities the agent selects for real tasks and adjust the playbook when you find ambiguous cases. In my framework, my subjective estimate is that incorrect activity selection happens in less than 1% of cases, possibly even below 0.5%, so this has not been a significant practical problem for me.
What I Learned
After working with both architectures, Prompt Engine with Context Compilation wins for me by a large margin.
The difference is not really about prompts or files. It is about deciding which responsibilities belong to an LLM and which belong to traditional software.
An LLM is excellent at understanding intent, interpreting ambiguous requirements, reasoning about code, and making decisions where the answer cannot be reduced to a simple algorithm. A script is excellent at following predefined dependencies, assembling files, validating inputs, and producing the same result every time.
Context Discovery asks the LLM to do some of both. A Prompt Engine lets me draw the boundary more deliberately.
That experiment eventually contributed to a broader principle I now use in my Semi-Automatic AI-Native Flow with a Human in the Loop.
For me, this is becoming one of the more interesting questions in AI-native development. We spend a lot of time asking how much more responsibility we can give to AI agents.
Maybe we should spend just as much time asking how much responsibility we can take away from them.
What edge cases did I miss here? How are you managing large instruction sets or context windows in your own AI frameworks? Let me know in the comments!
Top comments (0)