DEV Community: Nam Bok Rodriguez

Infinite Context Window for AI with Memory

Nam Bok Rodriguez — Mon, 27 Jul 2026 13:00:00 +0000

There is a number hiding in every AI coding setup that nobody looks at. It is the size of the instruction file, the CLAUDE.md or AGENTS.md or .cursorrules that gets pasted into the model's context on every single turn. Some of these files reach seventy kilobytes. That is roughly 17,500 tokens, paid again on every message, forever. And the reality we measured is simple: at that size, the model stops following the file anyway.

We ran an experiment to find out what actually happens when instruction files grow, and whether a memory database can replace them, using real public files rather than anything we wrote ourselves.

The setup

We took four agent instruction files from public repositories, copied verbatim and attributed:

The AGENTS.md from OpenAI's codex repository, about 22 KB
The AGENTS.md from Kiali, a service mesh console used with Istio, about 70 KB, the biggest file in the test
The AGENTS.md from Temporal, the workflow engine, about 8 KB
The AGENTS.md from Freerouting, a PCB routing tool, about 51 KB, held out completely, more on that below

For each file we picked rules that can be verified with plain string checks, so there is no judge and no interpretation involved. Did the output contain make lint, did it avoid interface{}, did the commit message carry a conventional prefix. Anyone can rerun these checks and get the same pass or fail.

Then a model performed real tasks from each repository's world under three conditions, five trials each:

The whole file in context. The best case for the file, the way most people run agents today.
MenteDB. The file is ingested once into memory. On each task the agent retrieves a few relevant rules, writes a draft, then retrieval runs again against what it wrote and the agent fixes anything it missed. Two passes, a few hundred to two thousand tokens total.
Nothing. No instructions at all. This is the control that keeps everyone honest: if the model passes a rule with no instructions, that rule proves nothing about retrieval.

The nothing condition scored between 22 and 40 percent. These rules cannot be guessed. Whatever compliance appears above that line comes from delivery.

The results

file	whole file in context	MenteDB	nothing
OpenAI codex, 22 KB	100% at ~5,600 tokens per turn	100% at ~1,700 tokens	27%
Kiali, 70 KB, the biggest	80% at ~17,500 tokens per turn	100% at ~2,200 tokens	40%
Temporal, 8 KB	89% at ~2,100 tokens per turn	100% at ~900 tokens	22%
Freerouting, 51 KB, held out	100% at ~12,700 tokens per turn	100% at ~2,200 tokens	22%

Read the 70 KB row twice. With the entire file sitting in context, the model followed 80 percent of the file's own rules. It had everything and still lost things. That is the quiet failure of big instruction files: they do not just cost tokens, they stop working. The 39 rules across all four files came through the memory path at 100 percent, at somewhere between 2.4 and 8 times fewer tokens.

This is what we mean by an infinite context window. It is not a bigger window, it is a window that never fills, because the cost per turn stays flat at a few hundred to two thousand tokens whether your rules weigh 8 KB, 70 KB or a megabyte.

The rule that proves the point

Here is a real trace from the 70 KB file's run. The task was small: add a Refresh button to a React component. The draft came back the way any model writes it.

<Button onClick={handleRefresh}>Refresh</Button>

Kiali has a rule that all user visible strings go through the t function for translation. Notice that the task never mentions translation, and neither does the draft in any word you could search for. No retrieval on the task description alone can find that rule, and we tried everything, including a reranker. The rule's relevance only exists in the output.

That is why the memory side runs two passes. After the draft, retrieval runs again against what was actually written. A button with a hardcoded label is exactly the shape the translation rule lives near, the rule comes back, and the revision wraps the label in t(). The whole exchange costs a couple thousand tokens. The seventy kilobyte file, carried on every turn since the beginning of the session, missed rules like this one anyway.

The same run caught the reverse failure too. Asked what to run before finishing a Go change, the model with the whole file in context answered with a broken merge of two commands. The memory side delivered the one rule that mattered and the answer was just make lint.

One row in the table deserves a special mention. The Freerouting file was held out completely: it never touched the system while we built it, its rule list was written before the first run, and it still scored 100 percent on the first try. The tasks are ours since someone has to write them, but the rules and checkers anchor to verbatim text from the files, and everything needed to rerun the benchmark or write your own tasks is public.

Try it

The whole benchmark, fixtures, mechanical checkers, parse caches and both runners, lives in the open source MenteDB repository under benchmarks/agent_file. Ingesting your own agent file takes one call, and files of any size now parse in the background:

from mentedb import MenteDB

db = MenteDB(api_key="...")
report = db.ingest_agent_file(open("AGENTS.md").read())
# 606 memories, 96 action triggers, 0 pinned to every prompt

Your rules stop being a tax on every turn and start being memories that show up when they matter, including the ones that only matter after the code is written. You can try it in your browser at demo.mentedb.com/agent-files: pick a real public agent file or paste your own, give the agent a task, and watch which rules arrive.

Your agent knows your preferences. It just never uses them

Nam Bok Rodriguez — Fri, 24 Jul 2026 20:15:19 +0000

There is a way for agent memory to fail that is worse than forgetting, and it took me a while to even recognize it as a failure. The preference is sitting right there in the store, and if I ask the agent what my commit style is, it answers correctly: single line, conventional prefix, no emojis. But the moment it actually fixes a bug and commits, the message comes out four lines long with an emoji in the subject, because knowing a rule and using it at the right moment turn out to be completely different problems.

Nothing here is technically broken, which is exactly why it hides so well. Memory lookups are driven by the conversation: embed what the user said, find similar memories. A rule about commit style shares no words and no meaning with the sentence fix the login bug, so the one moment the rule matters, right before the commit runs, is exactly the moment similarity search cannot find it.

There are two obvious ways out, and both are bad. You can pin the rule so it gets injected into every single turn, which works until you have thirty rules and every prompt drags all of them along, mostly irrelevant. Token bloat is the number one reason people rip memory systems out. Or you can put the rules in a static config file, which works until it does not: nobody updates it, it does not travel across machines, and when you change your mind the old rule just sits there.

The thing I eventually understood is that these preferences are not facts about topics at all. They are rules about actions. And rules about actions should be retrieved by the action, not by the conversation. You remember how your team writes commit messages when you are writing one, not when someone mentions login bugs. So the fix is a third way to look things up, next to the two everyone has: recall by topic for facts, a tiny set of rules that ride along on every turn, and now rules that fire on the action itself.

How it works

A rule gets a tag naming the action it governs: trigger:git-commit, trigger:pr-create. That is everything you do to store one. On the read side there is one call, recall_for_action. It looks the tag up in the index, checks who can see it the same way every other lookup does, drops anything superseded, and returns the rules newest first:

# a preference about HOW to commit, tagged with the action it governs
db.store(
    "commit messages: single line, conventional prefix, no emojis",
    memory_type="procedural",
    tags=["trigger:git-commit"],
)

# right before the agent runs a commit, its hook asks:
rules = db.recall_for_action("git-commit", k=6)
for r in rules:
    print(r.content)   # injected into context at the moment of the commit

The retrieval is deterministic: no embedding of the query, no similarity threshold to tune, no model call, just the action name as the key. That matters because this lookup runs at the worst possible moment to be slow or flaky, directly in front of the user's command.

The hook that asks at the right moment

The missing half is knowing when to ask, and the agent already knows: it is about to run the command. For Claude Code, setup installs a hook that looks at each command right before it runs, and when it is a commit or a PR, fetches the matching rules and drops them into context in that exact moment:

npx mentedb-mcp@latest setup claude-code
# installs a PreToolUse hook matched to Bash. Before a git commit or
# gh pr create, it fetches your trigger:git-commit / trigger:pr-create
# rules and injects them as context. Anything else: silence.

Most of the engineering here is in what the hook must never do. It never blocks or approves anything, it only adds context. It has a hard time budget, and on any failure, memory down, old server, timeout, it prints nothing and the commit runs untouched. A memory system that can break your git workflow gets uninstalled the same day, so the failure mode is always silence.

The parsing turned out to be where the real bugs hide. A rule firing on the wrong command erodes trust the same way a wrong memory does, so the matcher parses the command instead of grepping it: git -c commit.gpgsign=false commit fires, because the subcommand is commit. git config commit.gpgsign false does not, because the subcommand is config, even though the word commit appears twice. echo "git commit" does not, because it is quoted text. cd repo && git commit fires, because each segment of a compound command is checked on its own, and every one of those shapes has its own test in the suite.

Corrections work the way you would hope

These rules are ordinary memories, so everything from the last post about corrections applies to them too. Tell the agent you switched from squash merges to rebase, and the old rule is superseded: it stops showing up at commit time the moment you say so, and the history is still there if you ever ask what the old rule was. Rules also come back newest first, so if two versions of a rule are still around, the later one leads.

That is the part I care about most: the rule surfaces at the moment it applies, and the version that surfaces is the one you meant.

Try it

The engine call is recall_for_action in mentedb 0.27.2, the hosted API exposes it as the get_action_rules tool, and npx mentedb-mcp@latest setup installs the hook. Tag a commit rule, ask for a commit, and watch it show up at exactly the right moment. The live demo at demo.mentedb.com has the rest of the memory graph to explore.

Why your agent keeps repeating facts you already corrected

Nam Bok Rodriguez — Thu, 23 Jul 2026 13:00:00 +0000

Every agent I build eventually does the same thing. It repeats a fact I already corrected, stated with full confidence, as if the correction never happened. I tell it my coffee order changed from a black coffee to a latte, and a few turns later it hands me the black coffee. The memory didn't exactly fail. It retrieved something, it retrieved the version I had already replaced, because a raw vector store can tell you two facts are similar and has no idea that one of them replaced the other.

You can fix that, and the usual way is to put a language model in front of the store: on each new message it reads the text, compares it to what is saved, and decides whether to add, update, or delete. Corrections collapse into the current fact. But now it also means a model call and a network round trip on every single write, usually across a few services you now run. It works, it just costs you a model call every time you remember something.

That is the one of the parts I wanted to change. Not the retrieval, the deciding. Whether a new fact is a duplicate, a correction, a conflict, or something genuinely new is the real job of a memory, and I did not want that job living in code on top of a store, or using a model I have to call on every write. A model call per write is slow, it costs money, it pulls in more services to run, and it can answer a little differently on a bad day.

And most of these calls are not that subtle once you look at the text. A correction usually keeps the whole sentence and just swaps the value. That is a rule you can write down, not a judgment you need a model for. So I wrote MenteDB, a single embeddable engine in Rust, and I put the deciding inside it. When the latte arrives, it sees the same fact with a changed value and supersedes the black coffee. Same answer every time, no model asked.

What supersession does under the hood

The old fact isn't deleted. It gets a Supersedes edge in the graph, and recall stops returning it because superseded memories are excluded by belief propagation, so the history stays queryable while your prompt only sees the current value.

process_turn wraps this in one call: embed the message, run hybrid recall, store the turn, reconcile against what is already there, and return context for the next prompt. Here it is end to end using the API:

import os, requests

MENTEDB = "https://api.mentedb.com"
H = {"Authorization": f"Bearer {os.environ['MENTEDB_API_KEY']}"}

_n = 0
def turn(user, assistant=""):
    global _n
    _n += 1
    return requests.post(f"{MENTEDB}/v1/process_turn", headers=H, json={
        "user_message": user,
        "assistant_response": assistant,
        "turn_id": _n,
        "agent_id": "demo-user",
    }).json()

turn("My coffee order is a black coffee")
turn("Actually I switched to lattes")

r = turn("What's my coffee order?")
for m in r["context"]:
    print(m["content"])

Supersession is the sharpest example because it is obvious when it goes wrong, but it is one of many. Deciding that a memory has gone stale and should fade, that two are near duplicates and should merge, that one user's memories must never surface for another, that this handful is what is actually relevant to the current turn: those are the same kind of decision, made continuously, about what the agent should see right now. A raw vector store leaves all of that to your application code. The model on every write approach pays a model for it. I wanted it to be what the database does, so decay, deduplication, owner isolation, and relevance ordered recall are engine operations, not things you deal with.

One engine, not a stack of services

process_turn handles recall for you. It blends vector similarity with keyword search, so a query catches both the meaning and the exact words, and it puts the most relevant memories at the start and end of the context window, where the model pays the most attention. It only sends what changed since the last turn instead of replaying everything every message.

All of that is one Rust crate. Its own page based storage and WAL, its own HNSW index, its own graph. No external vector database to run alongside it, no separate graph store, no Postgres to manage. That is the part I actually care about, memory as a single dependency you embed, not a distributed system you operate.

Run it three ways, same engine

Hosted, if you just want a key:

export MENTEDB_API_KEY=mdb_your_key   # from app.mentedb.com
# then POST to https://api.mentedb.com/v1/process_turn (above)

Self host, one container, local embeddings bundled, no keys, nothing leaves the box:

docker run -p 6677:6677 -v mentedb-data:/data \
  ghcr.io/nambok/mentedb:latest

Embed the engine directly (Python over the Rust core):

from mentedb import MenteDB

db = MenteDB("./data", embedding_provider="candle")   # local, no key
db.store("The user prefers dark mode")
db.store("The user's editor is Neovim")

result = db.process_turn("what are their UI preferences?", turn_id=1)
for m in result.context:
    print(m.content)

Same engine underneath all three. The hosted product is that engine on managed infra.

Try it live

I created a demo here: demo.mentedb.com. Add facts, ask questions, and explore the memory graph they build.