There is a number hiding in every AI coding setup that nobody looks at. It is the size of the instruction file, the CLAUDE.md or AGENTS.md or .cursorrules that gets pasted into the model's context on every single turn. Some of these files reach seventy kilobytes. That is roughly 17,500 tokens, paid again on every message, forever. And the reality we measured is simple: at that size, the model stops following the file anyway.
We ran an experiment to find out what actually happens when instruction files grow, and whether a memory database can replace them, using real public files rather than anything we wrote ourselves.
The setup
We took four agent instruction files from public repositories, copied verbatim and attributed:
- The AGENTS.md from OpenAI's codex repository, about 22 KB
- The AGENTS.md from Kiali, a service mesh console used with Istio, about 70 KB, the biggest file in the test
- The AGENTS.md from Temporal, the workflow engine, about 8 KB
- The AGENTS.md from Freerouting, a PCB routing tool, about 51 KB, held out completely, more on that below
For each file we picked rules that can be verified with plain string checks, so there is no judge and no interpretation involved. Did the output contain make lint, did it avoid interface{}, did the commit message carry a conventional prefix. Anyone can rerun these checks and get the same pass or fail.
Then a model performed real tasks from each repository's world under three conditions, five trials each:
- The whole file in context. The best case for the file, the way most people run agents today.
- MenteDB. The file is ingested once into memory. On each task the agent retrieves a few relevant rules, writes a draft, then retrieval runs again against what it wrote and the agent fixes anything it missed. Two passes, a few hundred to two thousand tokens total.
- Nothing. No instructions at all. This is the control that keeps everyone honest: if the model passes a rule with no instructions, that rule proves nothing about retrieval.
The nothing condition scored between 22 and 40 percent. These rules cannot be guessed. Whatever compliance appears above that line comes from delivery.
The results
| file | whole file in context | MenteDB | nothing |
|---|---|---|---|
| OpenAI codex, 22 KB | 100% at ~5,600 tokens per turn | 100% at ~1,700 tokens | 27% |
| Kiali, 70 KB, the biggest | 80% at ~17,500 tokens per turn | 100% at ~2,200 tokens | 40% |
| Temporal, 8 KB | 89% at ~2,100 tokens per turn | 100% at ~900 tokens | 22% |
| Freerouting, 51 KB, held out | 100% at ~12,700 tokens per turn | 100% at ~2,200 tokens | 22% |
Read the 70 KB row twice. With the entire file sitting in context, the model followed 80 percent of the file's own rules. It had everything and still lost things. That is the quiet failure of big instruction files: they do not just cost tokens, they stop working. The 39 rules across all four files came through the memory path at 100 percent, at somewhere between 2.4 and 8 times fewer tokens.
This is what we mean by an infinite context window. It is not a bigger window, it is a window that never fills, because the cost per turn stays flat at a few hundred to two thousand tokens whether your rules weigh 8 KB, 70 KB or a megabyte.
The rule that proves the point
Here is a real trace from the 70 KB file's run. The task was small: add a Refresh button to a React component. The draft came back the way any model writes it.
<Button onClick={handleRefresh}>Refresh</Button>
Kiali has a rule that all user visible strings go through the t function for translation. Notice that the task never mentions translation, and neither does the draft in any word you could search for. No retrieval on the task description alone can find that rule, and we tried everything, including a reranker. The rule's relevance only exists in the output.
That is why the memory side runs two passes. After the draft, retrieval runs again against what was actually written. A button with a hardcoded label is exactly the shape the translation rule lives near, the rule comes back, and the revision wraps the label in t(). The whole exchange costs a couple thousand tokens. The seventy kilobyte file, carried on every turn since the beginning of the session, missed rules like this one anyway.
The same run caught the reverse failure too. Asked what to run before finishing a Go change, the model with the whole file in context answered with a broken merge of two commands. The memory side delivered the one rule that mattered and the answer was just make lint.
One row in the table deserves a special mention. The Freerouting file was held out completely: it never touched the system while we built it, its rule list was written before the first run, and it still scored 100 percent on the first try. The tasks are ours since someone has to write them, but the rules and checkers anchor to verbatim text from the files, and everything needed to rerun the benchmark or write your own tasks is public.
Try it
The whole benchmark, fixtures, mechanical checkers, parse caches and both runners, lives in the open source MenteDB repository under benchmarks/agent_file. Ingesting your own agent file takes one call, and files of any size now parse in the background:
from mentedb import MenteDB
db = MenteDB(api_key="...")
report = db.ingest_agent_file(open("AGENTS.md").read())
# 606 memories, 96 action triggers, 0 pinned to every prompt
Your rules stop being a tax on every turn and start being memories that show up when they matter, including the ones that only matter after the code is written. You can try it in your browser at demo.mentedb.com/agent-files: pick a real public agent file or paste your own, give the agent a task, and watch which rules arrive.
Top comments (1)
dog we gotta talk... ive been working on this for months......
i can do some crazy things... ive confirmed most of it.