DEV Community: Swapnanil Saha

The Bugs Only Dogfooding Finds: A Month Living Inside My Own Memory Daemon

Swapnanil Saha — Sat, 08 Aug 2026 17:11:39 +0000

There is a kind of bug you cannot find by testing. You find it by having to live with the thing: relying on it at two in the morning, following its advice while tired, restarting it in the middle of something that matters.

I build a tool called vectr. It is a small server that runs on my laptop and gives coding agents two things: search over a codebase, and a memory that survives past the end of a conversation. The part that makes this post possible is that the sessions which develop vectr use vectr as their memory. Findings about the daemon get written into the daemon. When it misbehaves, it misbehaves inside the work of fixing it.

Six defects came out of the last month that way. The test suite was green for every one of them, and it is not a small suite: roughly 3,900 tests, run in full before every push rather than sampled against the diff. I also run review agents over every branch before merge, deliberately pointed at the whole product rather than the change, and they did not flag any of these either. What found them was residency, by which I mean nothing more sophisticated than depending on the thing daily. Me, mid-task on something else, being annoyed.

This post is six war stories with their root causes and their fixes. It is also an argument about why a test suite cannot reach this class of defect, which turns out to be a more precise claim than "real usage is messy."

Part 1: The Setup

1. Living Inside the Thing You Are Building

Some vocabulary first, because the bugs are unintelligible without it, and none of it is hard.

A daemon is a program that runs in the background and answers requests. Vectr's daemon holds an index of my code and a store of notes, and it answers two kinds of caller. One is the CLI, the commands I type in a terminal. The other is my editor's AI agent, which talks to the daemon over MCP (Model Context Protocol), a standard for exposing tools to an AI assistant. Both callers reach identical logic underneath through separate thin translation layers, one per protocol, and those layers are separate code with separate bugs. That will matter in episode four.

The instance I actually live in serves nine project folders at once and runs in memory-only mode: notes and recall, no code index, no file watching. That mode exists because indexing is expensive and memory is not, a distinction episode six pays for in full. As I write this the instance holds 606 notes and has been up long enough that I no longer remember the exact command that started it. That last detail is the entire first bug.

Living inside a tool is different from testing it in three specific ways, and each one shows up below:

The state is old. A test starts from nothing. My daemon started weeks ago with flags I have since forgotten, on a machine that has been asleep, woken, filled, and cleaned since.
I am busy. Nobody dogfooding is dogfooding. They are doing something else, and the tool is in the way of it. That is when you follow instructions literally instead of thinking them through, which is exactly how a product's own advice gets to hurt you.
Nothing is staged. No fixture built the workspace, no fixture chose the port, no fixture decided where on disk the checkout sits. The environment is whatever accumulated.

Five of the six bugs below are vectr's. One is arguably mine, about my laptop rather than my code, and I have included it because the boundary it exposes is the same shape as the others and because pretending your hardware is not part of your system is how you spend three days blaming an API.

2. Why a Green Test Suite Proves Less Than It Looks

Here is the mechanical reason these bugs survived a suite that catches plenty of others.

Every test builds its own world before it asserts anything. A temporary directory with four files in it. A fake service object standing in for the real one. A port the harness picked. A clock that does not move unless the test moves it. This construction is not sloppiness, it is the whole point: a test is repeatable precisely because it controls everything the code touches.

The control is also the ceiling. A defect whose cause lives outside the constructed world cannot be caught by a test that does not construct it, and constructing it means having thought of it first.

I want to be careful here, because there is a cheap version of this claim that is wrong. Nothing stops you from writing a test that starts a daemon with one set of flags, shells out to restart, and asserts the mode survived. I wrote roughly that test after the fact, and it now guards the fix. So the barrier is not expressive power. The barrier is that the test encodes a hypothesis, and every one of the six below was invisible precisely because I did not hold the hypothesis. Residency does not give you tests you could not have written. It tells you which ones were worth writing.

The analogy that made this click for me

Tests are in vitro. A compound behaves beautifully in a dish and then fails in a body, because the body has a liver, a bloodstream, a kidney, and a schedule of meals. The dish has none of those, so the dish cannot possibly tell you about them.

A test suite has no liver. No operating system deciding to suspend your process, no second process holding a socket open from ninety seconds ago, no editor with a live connection it will not re-open, no memory of the flags you typed last month. Residency is in vivo: same compound, actual body.

So the useful question is not "did we test enough" but "what is on the other side of the wall the tests are built inside." For the six that follow, the answer is a specific boundary each time: process lifetime, packaging metadata, live client connections, transport adapters, the filesystem path the code happens to sit at, and the power manager. I will come back and lay them out side by side in section nine, once the stories have earned it.

Part 2: Six Bugs

3. The Remediation That Was the Hazard

I ran a routine command and the tool printed a warning at me:

vectr: daemon on port 8765 is running older code
(1.7.0+<sha> vs 1.6.0+<sha>) - run 'vectr restart <workspace>'

Set aside that the warning was wrong, which is the next section. I did what it said. I ran vectr restart <workspace>, exactly as printed, because that is what you do with a suggested command when you are in the middle of something else.

The daemon came back in full mode. It had been running memory-only for weeks. restart reads its mode from the flags you typed on that line and nowhere else, so a restart that omits --memory-only is a restart into the default, which is full indexing. Full mode on a nine-root workspace means walking every folder, cutting every source file into chunks, and computing an embedding for every one of them. On this laptop that is not a background task, it is an event. Section eight has the numbers.

I noticed within a couple of minutes because the machine got loud. There was measurable collateral too: the persisted chunk store went from 12,092 chunks down to 9,069 across that unintended full-mode interlude. My best guess is that the re-index rewrote the store under a different exclusion state than the one that had originally built it, but I have not chased that to ground, and I would rather leave it as an open loose end than dress a guess up as a finding. The notes were untouched, all 604 of them at the time, which is the only reason this was an annoyance rather than a disaster.

Warning: remediation text is an interface

Whatever your error message tells the user to run, they will run verbatim. Not a corrected version, not with the flags they used originally. Verbatim, while distracted, because that is the condition under which people read error messages.

Which means a suggested command is part of your API surface and inherits every obligation of one. If the safe invocation depends on state the user is expected to remember, you have shipped a trap and labeled it "help."

The interesting part is not that restart drops the mode. It is the shape of the failure: the product's own remediation advice was the mechanism of harm. Two independently defensible decisions composed into something neither of them was. The staleness banner suggests a restart. Restart takes its mode from what you typed. Nobody wrote a bad line of code, and the bug lives in the gap between two files.

What makes it slightly embarrassing is that the CLI already knows the answer. There is a helper in it that asks a running daemon for its current mode over the status endpoint, used elsewhere for exactly this kind of question. Restart could call it before killing anything, and does not. The state was one HTTP request away the whole time.

The durable fix has a precedent in the same codebase. When you decline editor config writes, that choice persists in a small marker file in the workspace so the tool stops asking. Mode, port, and host should persist the same way, and restart should reuse the persisted config unless you explicitly override it. Until that lands, the banner should print the mode-complete command rather than the bare one, since it just talked to the daemon in order to produce the warning at all.

What survives a restart, as shipped: port (yes, since the port fix in episode three), roots served (yes, from the workspace file), mode (no). The one flag that decides how much CPU the next hour costs is the one that does not carry across.

4. The Version Check That Cried Wolf Backwards

Now the warning itself.

That check exists for a genuine problem. Most of what you see from vectr is rendered by the daemon, not the CLI, and the CLI imports the working tree fresh on every invocation. Upgrade the source while a daemon from last week is still running and you get a silent split: new CLI, old rendering, no error anywhere. I hit that in early July and could not see any of my own UX changes. The fix was a shared stamp, computed identically on both sides: the package version plus the short git sha when running from a checkout, stamped into the daemon at startup and exposed in its status response.

Then it started lying to me. The banner said the daemon was running older code, and printed 1.7.0+<sha> for the daemon against 1.6.0+<sha> for the CLI. Same seven characters of sha on both sides. The daemon's version was the higher one. Every part of that sentence contradicts the other parts.

Here is what happened. The version component comes from installed package metadata, and vectr is installed in editable mode, which means the metadata was written once when I ran the install and has been sitting there ever since. The project config had moved to 1.7.0. The metadata still said 1.6.0. The daemon, started after a reinstall, had picked up the newer metadata. So the CLI compared its own stale label against a fresher one and reported the difference as the daemon being behind.

Insight: inequality is not order

Two strings that differ tell you they differ. They do not tell you which came first, and no amount of string comparison will make them. The check had one component carrying genuine code identity, the git sha, and one component that is a label a human types into a file and a packaging tool caches at install time. It compared the whole thing, then described the result with a word, "older," that only an ordering could justify.

Write the predicate out and the error is obvious:

What the three questions actually require

is_different = sha_daemon != sha_cli

That is the only one you can answer from the stamps alone, and it is the one the feature exists for. "Is the daemon running my code" means "is it running my commit," so the version prefix has no business in the comparison at all.

is_older = git merge-base --is-ancestor sha_daemon sha_cli

Ordering is a question about history, not about text, so answering it costs a git call and can fail outright when the two commits sit on diverging branches. Which is a real state, and one where no honest answer to "which is older" exists.

The shipped check computed the first question over the wrong operand and then reported the second one without ever asking it.

So the fix is to compare the sha and to word the message as a version mismatch whenever only the label differs. Never claim "older" from a lexical difference.

What makes this worth more than a paragraph is the second-order damage. A false alarm does not cost you one wasted minute. It costs you the alarm. I now have a check that fires on a condition I know to be benign, which trains me to skim past it, which means it will not work the day it is right. And it caused real harm through a path nobody designed: it told me to restart, and the restart was the first bug. A false positive in a warning is not a cosmetic defect when the remediation attached to it is expensive.

5. The Restart That Orphaned the Session

Same restart, different casualty. The editor session I was working in lost every vectr tool. Not erroring when called, which would at least have been a clue: simply absent from the session's tool registry, as though the server had never offered them. They never came back. A new session in the same editor had all of them.

Meanwhile the daemon was completely healthy. I could hit its HTTP API by hand and get correct answers to the same questions the vanished tools would have answered. The server was fine, and what had broken was the relationship between it and a client that no longer existed to be told about it.

One mechanism behind this got caught and fixed on 29 July, and it is a good example of how ordinary the cause of a total outage can be. Vectr picks its port through a small registry: when a workspace has a previous port recorded, reuse it, specifically so that already-written editor config files stay valid. The reuse check probes whether the port is free by trying to bind it. A socket that was just closed sits in TIME_WAIT for a while afterwards, and a plain bind attempt against a TIME_WAIT socket fails. So the probe reported the previous port as busy, in exactly the window that every restart creates, and the port walk moved the daemon to 8766 while every config file on disk still said 8765.

Reproduced live: stop, start one second later, daemon bound to 8766. The editor hooks kept working the whole time, because they look the port up in a registry file instead of trusting a config value, which is why the outage looked partial and confusing rather than total. The tool surface broke with no error anywhere naming the cause. A second defect compounded it: the config writer reported that it had updated the settings file for every folder, and the file still read localhost:8765, because the port handed to the writer was not the port that ended up bound.

The fix landed as four changes, all deterministic:

The free-port probe sets SO_REUSEADDR, matching what the actual server bind does, so TIME_WAIT reads as free.
Port selection retries the previous port a few times with a short delay before it ever walks upward.
Vectr's own entry in an editor config file is always rewritten to the port that was actually bound, while every other key in that file stays merge-only-add.
Start and restart compare each known config file against the real bind and print a warning naming the stale files.

And now the honest part, because the port fix does not close this. The client I use does not re-handshake after it loses a connection. Nothing in the transport obliges it to, and other clients may well retry, but from mine the tools stay gone for the rest of that session even with the port preserved, because the process it completed its handshake with no longer exists. Restarting a server is cheap for the server and expensive for whoever was mid-sentence with it.

Warning: restart is a client-facing operation

A daemon's restart story usually gets designed from the daemon's point of view: come back up, reload config, re-bind, resume. That is the easy half. The half that decides whether a user curses at you is what happens to the callers that were mid-conversation with the old process.

A server cannot force a client to reconnect. What it can do is need restarting less often, say out loud what a restart will cost before doing it, and document a fallback surface instead of leaving each user to discover their own.

The detail I keep coming back to is that I did not know the HTTP fallback was usable until the moment I needed it, and I wrote the thing. An undocumented escape hatch is an escape hatch for exactly one person.

6. The Mock That Lied Politely

This one is older than the others, from mid-June, and it belongs here because it is the purest example of a test suite validating a fiction.

Symptom: the HTTP endpoint that answers "where is this symbol defined" returned a 500 for every symbol, in every language, every time. Not a rare path or an unlucky input, the whole endpoint, dead on arrival for as long as it had existed. The test suite was green.

Cause, in one line: the service method returns a LocateResult wrapper object, and the route iterated it as if it were a list.

# app/routes.py

# before
symbols = svc.locate_with_snippets(body.name, limit=body.limit)
...
    for s in symbols        # TypeError: 'LocateResult' object is not iterable

# after
result = svc.locate_with_snippets(body.name, limit=body.limit)
...
    for s in result.symbols

Now the part that matters. Why did nothing catch it?

Two reasons, and they reinforce each other. First, there were no tests for that route at all. The MCP path to the same feature was tested and happens to be unaffected, because it hands the whole result object to a formatter instead of iterating it. Somewhere along the way "the feature is tested" quietly became "every way of reaching the feature is tested," which are different claims once two translation layers sit over one core.

Second, the shared service mock that every API test uses was configured like this:

# tests/test_api.py, the fixture

# before: a type the real service never returns
svc.locate_with_snippets.return_value = []
svc.format_locate.return_value = "No results."

# after: the real wrapper, with a real symbol inside it
svc.locate_with_snippets.return_value = LocateResult(
    symbols=[Symbol(name="PyDict_New", kind="function",
                    file_path="Objects/dictobject.c",
                    start_line=812, end_line=824, ...)],
    resolution_strategy="exact", query="PyDict_New",
)

A bare list. Convenient, empty, and a shape the real function has never returned in its life. Any test that had touched the route would have iterated that list happily and passed, because iterating a list is exactly what the buggy code does correctly.

Warning: a mock is a claim about someone else's code

When you write return_value = [] you are not simplifying. You are asserting, silently and with nothing in the language or the tooling able to check it, that the real function returns a list. If that assertion is wrong, every test depending on it exercises a program that does not exist. That is worse than no coverage, because it reports as coverage.

The obvious objection is that Python's mock library will build a mock from the real object for you, and that specced mocks are the standard answer to exactly this. Worth being precise about what that buys: auto-speccing constrains which attributes exist and what call signatures are legal, so it catches you calling a method that was renamed or passing an argument that does not exist. It says nothing whatsoever about return types. return_value remains whatever you assign, and no framework I know of will tell you that the real function has never once returned that shape. Which leaves one reliable discipline: capture a real payload once, and build the fixture from it.

It happened again three days later in a different layer, which is what convinced me this was a class rather than an incident. A benchmark harness reported that every one of its hook injections had delivered zero content. That reads as a serious product failure, so I spent an afternoon investigating a product that was working fine. The metric was the broken part. The real event emitted by the agent CLI carries the hook's output as a JSON string inside a field, and the parser only walked nested dictionaries, so it never looked inside the string and therefore always found nothing. It had been green the entire time because the test mock fabricated a nested-dictionary shape the CLI does not emit. Injection had in fact worked: 7,404 characters delivered live. What it cost was a full paid benchmark run whose headline number was fiction.

Three rules came out of that pair, and they have held up:

Mirror the real return type, not a convenient stand-in. If you are mocking a wire format, capture one real payload first and build the mock from it.
Every externally reachable route gets its own test. Green on one transport proves nothing about another when the adapters are separate code.
When a metric reads zero on a path you believe works, suspect the measurement before the product. A measurement bug looks exactly like the failure it is measuring, and it is cheaper to check.

The sequence that fixed it is worth spelling out, because the ordering is the lesson. Step one: fix the fixture so the mock returns the real type, and add the route test that never existed. The suite goes red against completely unchanged production code. Step two: fix the route to unwrap result.symbols. Green again, and now pinned. The faithful mock is what made the bug visible; the code fix was the easy part that followed.

7. The Test That Was Really Testing My Filesystem

I run coding subagents in isolated git worktrees. Two integration tests failed in those worktrees and passed on the main checkout. The failures looked like a regression from whatever the agent had just changed. They were not.

The tell was the clock. The failing test reported recall of 0.00 in about a third of a second, where a passing run takes fifteen. Nothing that searches a real index comes back in 0.34s. Nothing had been indexed at all, so precision and recall were trivially zero, and the test dutifully reported a search-quality score instead of the far more useful fact that its input was empty.

My first explanation was a good one, which is exactly how it became lore. Those worktrees live under a dot-prefixed directory, and the indexer prunes hidden directories during its walk:

# agent/indexer/_core.py
dirnames[:] = [d for d in dirnames
               if d not in all_excluded and not d.startswith(".")]

Skipping hidden directories is correct: it keeps version control internals, virtual environments, and caches out of a code index. And it does explain that particular case. As the general rule it got treated as, it turned out to be wrong, which I only found out by bothering to run a control.

A three-way control run knocked out the hiddenness story as the general cause. The replacement hypothesis, "worktrees are the problem," was confounded in a way I find genuinely funny in retrospect: both non-hidden control worktrees had been created under /tmp, which macOS resolves to /private/tmp, and tmp is an unanchored entry in vectr's own ignore file. Every supposedly clean control was excluded too, by a different mechanism, producing an identical symptom. The decisive probe was a worktree at a path with no tmp component and no dot-directory: three tests passed in 16.23 seconds.

Underneath the folklore was a real product bug, nastier than the test failure that led me to it. The function deciding whether to index a file was passing the absolute path to the ignore-pattern matcher, while the two checks directly above it correctly used the workspace-relative path.

That matters because of what an ignore pattern means. A pattern in an ignore file is defined relative to the directory containing that file. An unanchored entry like tmp/ matches a directory named tmp at any depth below that root, and by definition it can never say anything about what sits above the root, because the file that declares it has no jurisdiction there. Hand the matcher an absolute path and you have quietly extended its jurisdiction to your entire filesystem, so the pattern starts matching ancestors that the repo owner never had an opinion about.

Insight: what this meant for anyone who is not me

If your repo lived under a directory named tmp, build, env, dist, cache, or node_modules, and your own ignore file listed that name, vectr indexed zero files. Silently. The index call returned zero files and zero chunks, and zero is what success looks like when there was nothing to do.

Nobody would have reported this as "the ignore matcher uses the wrong string." They would have reported "search finds nothing" and I would have asked for their query.

Fixed on 27 July at both call sites, the bulk walk and the file watcher, which now pass the relative path to the shared predicate. There was an intended side effect: path-scoped patterns like docs/* now match workspace-relative paths the way ignore-file semantics say they should. Before the fix they could never match anything.

The remaining work is not in the product, it is in the test. A test whose precondition is "the index is not empty" should assert that precondition and skip with a reason that names the matched pattern and the offending path component, rather than reporting recall 0.00 and inviting the next person to spend a control run rediscovering this. Environmental preconditions deserve the same care as assertions. If they stay implicit, the first plausible story about a failure becomes team knowledge, and team knowledge is very hard to unlearn.

8. The Laptop That Ate the Fleet

Long agent runs kept dying mid-stream. The error said the stream had stalled or the connection had failed, so I filed it under network flakiness on the API side and re-ran things. Roughly forty kills accumulated across four days that way.

Then I did the thing I should have done on kill number two: lined the timestamps up against the machine's own power log.

pmset -g log | grep -E 'Wake|DarkWake'

Every kill matched a wake event to the second. Five or more exact matches in a row, and every one of the forty consistent with the pattern. That is not correlation you argue with.

The mechanism is unglamorous. Idle sleep on battery, or closing the lid, suspends the entire process tree. The connection to the API dies while everything is frozen. On wake, the ten-minute stream watchdog notices immediately that nothing has arrived and records a kill. The stack was never broken. It was asleep, and it got blamed for being unreachable while it was.

Insight: "the run failed" is not a diagnosis

An error message describes the symptom at the layer that noticed. A watchdog that fires on a dead stream cannot tell you whether the stream died from a network partition, a server fault, or the operating system putting your process to sleep. Those need different fixes, and only one of them is in your code.

Practical resolution: caffeinate -is for the duration of any long run. The -i is the flag doing the work here, since it blocks idle sleep, which is the one that fires while you are away from the machine; -s blocks system sleep and only has effect on AC power. Neither can override closing the lid on battery, so it comes down to lid open or plugged in. On AC there is a system setting that prevents automatic sleeping when the display is off, and that is the durable version of the fix. One useful non-symptom: display sleep alone is harmless, so "the screen went dark and the run died" is a misleading intuition. Only system sleep kills.

The second half of this episode is about load rather than sleep, and it is why full-mode indexing counts as a hazard back in episode one. During one corpus re-index I watched the daemon hold 329% CPU for about two and a half hours, with system load at 21.24 and swap at 19.0 of 20.0 GB on a machine with 16 GB of RAM. Nothing else on the laptop was usable for the duration. The cost sits entirely in the search-index lane: roughly 350,000 embeddings for that corpus, split between one vector per chunk of code and a second pass producing a vector per symbol from its signature and docstring. Memory mode costs close to nothing by comparison, which is why the always-on instance runs unnoticed for weeks.

One aside that cost me an hour of confusion: processes launched from the editor's terminal are children of the editor's process tree, so the activity monitor attributes their memory to the editor. A 32 GB reading next to the editor's name was my own Python stack, not the editor being a memory hog.

Before the obvious suggestion: simply running the indexer at a lower scheduling priority does not fix this. Priority governs who gets the CPU, and by the time swap is at 19 of 20 GB the contended resource is memory, where being polite about CPU buys you nothing. A governor has to pace the work and cap concurrent batches, not just ask nicely for fewer cycles.

None of this is a vectr bug in the narrow sense. It became product work anyway, because the version a user hits is strictly worse than the version I hit: they install the tool, it starts indexing, their machine becomes unusable, and they uninstall. Three items came out of it. An index resource governor with a CPU budget, batch pacing, and pause-resume. Deferring the second embedding pass on large repositories. And the strategic one, that memory-only should be the default install, with search indexing an explicit per-workspace opt-in that shows a cost estimate before it starts. The entry price of a memory tool should be close to zero.

Warning: the OS is part of your system

On a server you can assume the machine stays awake and the scheduler is roughly fair. On developer hardware neither holds. The power manager will suspend you, thermal limits will throttle you, and a second heavy process will starve you. If your software takes hours and your users run it on laptops, the power model is a component you depend on, whether or not you have modeled it.

Part 3: The Pattern

9. Six Boundaries a Test Suite Cannot Cross

Laid out together, these are not six unrelated mistakes. Each one sits on a specific boundary between the program and something a test harness constructs, replaces, or ignores.

Episode	The boundary	Why a test cannot represent it
Restart drops the mode	One process lifetime to the next	The test starts the daemon it tests. It can never be surprised by a flag typed weeks ago, because it typed the flag.
Version check reversed	Packaging metadata vs code identity	In a test both stamps are computed in one process from one checkout. Metadata staleness needs an install that happened at a different time.
Orphaned MCP session	Daemon vs its live clients	The harness is the only client, and it is created after the server. There is no already-connected editor holding a session that outlives a restart.
Mock returned the wrong type	Two transports over one core	The mock is the boundary, so it cannot check itself. Coverage on one adapter says nothing about the other.
Ignore matched the absolute path	Code vs where it sits on disk	Fixtures live in a temp directory whose absolute path nobody chose or examined. The bug needs a specific ancestor name.
Sleep killed long runs	Process vs the OS power model	Tests finish in milliseconds and the harness never sleeps. Suspension is not an event the suite has any way to produce.

Three of these are not "the code is wrong" in the usual sense. The restart does what its arguments say. The ignore matcher matches the string it is given. The daemon binds a port that is genuinely free. In each case the defect is a broken contract with something outside the process, and the code inside the process is a faithful implementation of the wrong agreement.

Which suggests a practical filter for where to look. Ask what your tests replace with a stand-in, and what they construct fresh every run. Both lists are lists of blind spots. Mine were: the environment the daemon was started in, the identity of the code, the client on the other end, the service behind the mock, the absolute path of the workspace, and the operating system. Every bug in this post is on that list, and I could have written the list before finding a single one of them.

Insight: the cheapest audit I know

Take one integration test and write down every value it invents: the directory, the port, the clock, the fake service, the flags, the machine state. That inventory is the set of assumptions your suite is structurally unable to question. You do not need residency to produce that list. You need residency to find out which items on it are actually wrong.

10. What Dogfooding Is Bad At

Six good catches makes a persuasive post, so here is the other column, because a method whose limits you cannot state is not a method.

I am the worst possible user. I know every workaround before I reach the wall that needs it, and I never read the documentation because I wrote it. More to the point, I have not experienced a first install in months, and the first ten minutes is where most people decide whether a tool is worth keeping. The famous version of this bias is teams on fast machines missing the performance problems ordinary users hit daily. Mine runs the other way: my laptop is underpowered relative to the corpora I index, so I feel resource pain acutely and am completely blind to the fresh-start experience I never repeat.

One machine, one operating system, one workflow. All of the above is macOS, on a single multi-folder workspace with one particular ignore file in it. Residency samples very deeply from one point in the space and says nothing at all about the shape of the distribution around it.

It finds classes, not rates. I can tell you the mock-fidelity failure exists and reproduces trivially. I cannot tell you how often users hit any of this, and any number I offered you would be made up. Rates need telemetry or a user base, and I have neither at a scale where a rate would mean anything.

The suite is not the villain here. It was green through all six of these and it is still the only reason I can change any of this without fear. The three instruments answer different questions. A test pins behavior you already understand so it stops moving. A reviewer interrogates the code you wrote. Residency is the only one of them that samples the assumptions nobody ever wrote down, which is also why it cannot be scheduled or assigned. Lean on residency alone and you land in the same place as leaning on tests alone: excellent coverage of one region, confident silence about everything else.

There is also a trap specific to this setup, which is that vectr is the memory of the sessions that build vectr. When the memory layer misbehaves, it degrades the context of the work fixing it. That is a real coupling and it is not all upside. It does mean I notice immediately. It also means my judgment about severity is made by someone whose current session state is affected by the bug, which is not a neutral vantage point. I have not found a good answer to that beyond writing findings down before acting on them.

11. The Rule I Needed Telling Three Times

None of the above matters if you route around it.

The rule is simple to state: a bug found while dogfooding is a product task. Fix it, or write it down with the end-user impact named. Never silently work around it. I was told this three times before it stuck, and I want to be precise about why it took three.

The first two times I agreed with it in principle and kept doing what everyone does, which is notice a rough edge mid-task, work around it in two seconds, and carry on with the thing I was actually doing. The workaround is always cheaper right now. That is the whole problem: the cost is real but deferred and lands on someone else. The third time it came with the sentence that made it stick. Roughly: it is not only about you, users are going to hit this too.

Which reframed the workaround as a decision rather than an omission. If I know the tool tells you to run a command that will melt your laptop, and I quietly stop running that command, I have not avoided a bug. I have decided that everyone who does not know what I know should hit it.

Insight: friction is the measurement

The moment your own tool annoys you is the only free signal you will ever get about that defect. You are already in the failing state, with full context, with the cause fresh. Working around it spends that signal on getting your task done two minutes sooner.

Dogfooding as a source of anecdotes is worth very little. Dogfooding with a contract that converts every surprise into a fix or a written item is a QA instrument. The difference is entirely in what happens in the sixty seconds after the surprise.

Mechanically, what makes it stick for me is three things. A place to put findings that is not my memory, with the user impact spelled out rather than a one-line reminder I will not understand next week. A default of fix-first for anything on the surface a new user meets in their first ten minutes. And a release rule that the tail does not begin while the discovered-issue list has items on it, which stops "we will get to it" from being a decision made by silence.

Two of the six bugs here are still open items rather than merged fixes, and I would rather say that than imply a clean six-for-six. They are written down with their impact, which is the deal. The distinction that matters is not fixed versus unfixed. It is filed versus forgotten.

Summary

I opened by saying there is a kind of bug you find by living with software rather than testing it. The six here say something sharper than that. They are not random messy-reality bugs. Every one of them sits precisely where my tests construct their own world and the real world differs: the process that started last month, the metadata that lagged the code, the client that was already connected, the transport nobody tested, the path on disk, the power manager.

That list is derivable. You can write it for your own system this afternoon, from an inventory of what your fixtures invent and what your mocks replace, without finding a single bug first. Residency is what tells you which entries on the list are currently costing you.

Any command your error message suggests must be safe verbatim. If the safe form depends on state the user is supposed to remember, persist that state yourself. Following the product's own advice should not be the hazard.
Compare identity, not labels. A git sha is what the code is; a version string is what someone typed and a packaging tool cached. And never infer "older" from two strings merely differing.
A daemon with live clients needs a restart story for the clients. Coming back up cleanly is the easy half; the callers mid-conversation with the old process are the half users feel.
A mock that returns the wrong type is worse than no test. It reports as coverage while validating a program that does not exist. Capture a real payload before you fabricate one, and give every externally reachable route its own test.
An empty result that reports success is the worst failure mode there is. Zero indexed files looked exactly like nothing to do. Make pipelines that can produce zero justify it out loud.
Your users' hardware is part of your system. Sleep, thermal limits, and one heavy process starving another are all inside your failure domain if your software runs for hours on a laptop.
Convert friction into fixes or the whole exercise is just anecdotes. The workaround is always cheaper in the moment, and it silently decides that your users should hit what you just dodged.

The parts of vectr I trust most are not the parts with the best test coverage. They are the parts I have been unable to break while depending on them for months, which is a weaker guarantee in theory and a much more convincing one in practice. If you build tools, the cheapest QA available to you is to need your own tool badly enough that its failures cost you something, and then to be honest about what it costs.

Agent Memory Needs a Trust Ladder: Provenance, Revocation, and Notes That Lie

Swapnanil Saha — Fri, 07 Aug 2026 14:00:19 +0000

Somewhere in a project I work on, an agent once recorded that a particular lock was released when a function returned. It was not. The lock was released when the surrounding scope exited, which in that code path was several frames later. The note was wrong the second it was written, and it was written in the crisp declarative voice that every good note is written in, so it read like fact for weeks.

Two words are about to do a lot of work, so let me pin them down. A session is one continuous run of an AI coding agent: you open it, you work, it ends, and everything it figured out evaporates unless something wrote it down. Memory is whatever writes it down, so tomorrow's session starts where yesterday's finished rather than at zero.

Failure in that second thing is what nobody builds for. We have spent two years getting good at storing and fetching: embed the note, rank it by similarity, inject it at the right moment. All of that assumes the note is true. But a note is a claim, made by a fallible process, at a particular moment, about a world that keeps moving, and a memory system whose entire interface is save and search has quietly declared every claim equally true, forever.

This post is about the third verb. Not how to store a note or how to find it, but how to say how much this note is worth when it comes back. I will walk through the trust model I shipped in vectr, my working-memory and code-search layer for AI coding agents: provenance classes, promotion, revocation that leaves a body behind, and staleness that flags without accusing. Then I will spend the last third of the post on the parts I got wrong or have not solved, because those are more useful to you than the parts that work.

Part 1: The Problem With Two Verbs

1. Save and Search Is Not a Memory System

Take the minimal agent memory store. It has two operations. save(text) writes a string somewhere durable. search(query) returns the strings most similar to the query. Almost every memory layer shipping today is this, plus engineering: better chunking, better ranking, a graph instead of a flat list, a summarizer that compacts old entries.

Now watch what happens over six months of real use. The store accumulates a few hundred notes. Some are excellent. Some were written by a model that misread a stack trace. Some were true in March and false by June because someone refactored the module. A handful are the user's own words, typed in frustration, and worth more than everything else in the store combined.

When search returns five of them, all five arrive as the same kind of object: a paragraph of confident text. The reading agent has no way to tell the user's standing instruction from a machine's half-formed guess about a stack trace. It will read them in ranked order and treat them as equally authoritative, because nothing in the payload says otherwise.

The core asymmetry. Retrieval quality and trust quality are independent. A perfect retriever that surfaces exactly the right note still hands the agent a lie if that note was wrong when written. Improving similarity ranking does nothing for this failure. It just delivers the wrong belief faster and more reliably.

The standard answer in the field is conflict resolution at write time: when a new fact contradicts an old one, the newer wins. Mem0 runs a model-driven pass at ingest that decides whether an incoming fact adds, updates, or deletes an existing one. Temporal knowledge graphs like Graphiti do a more structured version with validity windows on edges. Recency is a decent heuristic and I use a form of it, but look at what it cannot reach. It says nothing about a note that was wrong at birth, because nothing later contradicts it. It says nothing about whether a note is the user's own directive or a model's paraphrase, because both are just text. And when the newer fact is the wrong one, recency actively hurts: the store overwrites a correct belief with a fresh mistake and reports no conflict, because from its point of view nothing went wrong.

There is a sharper version of that last problem. The arbitration itself is usually done by a model, which means the decision to overwrite is produced by the same kind of process that produces wrong notes in the first place. You have not removed the failure. You have moved it one layer down, where it is harder to see, because now the loss is silent: the old note is gone and nothing records that a judgment was made.

What is missing is not a better arbiter of which claim wins. It is a record of provenance (the recorded origin and chain of custody of a piece of data) and lifecycle attached to the claim itself, so the agent reading it can weigh it instead of assuming it.

2. Three Ways a Stored Note Lies

Once I started auditing notes rather than just counting them, the failures sorted themselves into three groups. They need different machinery, which is why lumping them together as "hallucination" or "drift" gets you nowhere.

1. Wrong when written

The agent misunderstood the code and stored the misunderstanding. This is the lock example from the intro. There is no contradiction to detect, no timestamp that helps, and no later event that flags it. The note is internally consistent, well-formatted, and false. The only thing that catches it is a later session doing the work again and noticing the mismatch, and by then the note has been read a dozen times.

What makes this one nasty is that the note's confidence is a property of the writing style, not the evidence. Models write notes in the same register whether they traced the call graph carefully or skimmed one file. There is no linguistic signal to key on. Any system that tries to infer confidence from the note's wording is reading tea leaves.

2. Right then stale

The note was accurate. Then someone moved the function, changed the default, renamed the config key, and the note kept its confident present tense. This is the failure everyone already knows about, and it is the easiest of the three to detect, because the world leaves fingerprints: file hashes change, symbols move, mtimes advance.

It is also the one most commonly over-corrected. A changed file does not mean the note is wrong. Over a busy month I watched one file take dozens of commits without invalidating a single note attached to it, because every one of those commits landed in a function the notes never mentioned. If your staleness detector turns "this file changed" into "this fact is now false," you have swapped one lie for another.

3. Misattributed authority

This is the one nobody designs for, and the reason I wrote this post. A note says: always run the test suite with the virtual environment's Python, never the global one. Who said that? If the user said it, it is a standing rule and the agent should follow it without deliberation. If a model inferred it from one flaky test run, it is a plausible guess that deserves a check. Same words. Completely different weight.

The failure runs in both directions, which is what makes it hard. An agent's paraphrase can end up wearing the user's voice, in which case the system over-trusts a guess. Or a genuine user instruction gets stored through an agent-mediated path and comes back hedged, in which case the system under-trusts the one thing in the store that was never in doubt. I come back to both directions in section 8, because I have not fixed them.

Laid out side by side, the reason these need separate machinery is that they leave completely different amounts of evidence behind:

Failure	Evidence available to the system	Usual answer	What it actually needs
Wrong when written	None. Nothing about the note or the world indicates it.	Nothing, or a self-reported confidence score.	A later reader to catch it, and a way to record the catch so it sticks.
Right then stale	Plenty. Hashes, mtimes, moved symbols.	Recency ordering, or expiry after a fixed age.	A screening signal that says verify, not one that says false.
Misattributed authority	Only at write time, and only if you capture it then.	Nothing. Every note is the same kind of text.	A provenance class stamped at write and rendered at read.

The middle row is the one the field has mostly solved. The other two are open, and the third is not even widely recognised as a problem.

The lab notebook analogy. A research lab does not keep one undifferentiated pile of statements. It keeps a notebook where every entry is dated and initialed, results get countersigned by someone who reproduced them, and a retracted result stays in the notebook with a line through it and a note about why. Nobody tears the page out. The struck-through entry is doing work: it stops the next person from running the same doomed experiment.

That is the whole design. Dated, initialed, countersignable, struck through rather than erased. The rest of this post is what those four things look like when the reader is a model rather than a graduate student.

Why a confidence score does not work here. The obvious move is to have the writing agent attach a confidence number. It fails for the same reason the writing style fails: the number is produced by the same process that produced the possibly wrong note, and models are poorly calibrated about their own reasoning. A self-reported 0.9 on a misread stack trace is worse than no number, because it launders a guess into a measurement. Provenance is different: it records a structural fact about how the note came to exist, which the system knows independently of what the agent believes.

Part 2: What I Shipped

3. The Trust Ladder: Three Provenance Classes

Every note carries one value from a three-element ordered vocabulary. Not a score, not a float, not anything a model computes. A closed enum (a field that accepts only values from a fixed, predeclared list), checked at write time, answering one question: how much reviewing judgment stands behind this?

Class	Meaning
human	A person recorded or endorsed this. The only class that ever renders as an unhedged instruction. Not settable at write time by any agent-facing call.
agent	An AI session deliberately recorded this after doing some work. The default. Real judgment was applied, by a fallible judge.
auto	Captured by a mechanism with no reviewing judgment at all. A git hook that notices a commit, for example. Nobody looked at it.

The ordering is auto below agent below human, and the ladder is not decorative. It decides three things: how the note is framed when it comes back, whether it is allowed to be a standing rule at all, and how far it can be raised later.

The framing is the interface

Provenance would be useless as a database column alone. The agent reading a recalled note never sees your schema. It sees a block of text in its context window, so the class has to be spent on the wording of that text. Each class maps to a fixed prefix, prepended to the note's content on the recall path. These are the literal strings, copied out of the shipped code:

human + directive  →  "DIRECTIVE (standing rule from the user — follow it): "
human + any kind   →  "Recorded by the user: "
agent  (default)   →  "Memory to verify (recorded by an AI session, not human-endorsed): "
auto               →  "Auto-captured (weakest confidence, no reviewing judgment
                        applied — verify before relying on this): "

Only the top of the ladder gets an imperative. Everything else is framed as something to check. That asymmetry is deliberate and it is the entire point: the default posture toward a stored note is skepticism, and confidence is something a note has to earn its way up to.

Immutable at write, and a pure function. The prefix is computed from the note's stored (provenance, kind) pair and nothing else. Not from the query, not from how the note is phrased, not from a model's read of it. That means the framing is reproducible: the same note renders identically on every recall, on every machine, forever, until someone explicitly promotes it. A trust signal that varies by context is not a trust signal.

The write-time enum is where the security lives

One detail turns the ladder from labelling into something with teeth. The provenance parameter exposed on the agent-facing tool accepts exactly two values:

"provenance": {
  "type": "string",
  "description": "How much to trust this note when it resurfaces (default
    'agent'): 'agent' = self-recorded, framed as memory to verify; 'auto' =
    no reviewing judgment, weakest framing, incompatible with
    kind='directive'. 'human' is only reached via explicit promotion,
    not settable here.",
  "default": "agent",
  "enum": ["agent", "auto"]
}

An agent can declare itself weaker than the default. It cannot declare itself stronger. human is not in the enum, so the strongest class on the ladder is unreachable from inside the agent's own control loop, by construction rather than by policy. There is no prompt that talks an agent into marking its guess as human-endorsed, because the field will not accept the value.

The same principle produces one more restriction. A note with kind="directive", meaning a standing rule that gets injected at the start of every future session, is rejected at write time if its provenance is auto. Nothing looked at it, and now it would sit in front of every session forever. The store raises rather than storing it:

ValueError: provenance='auto' is not allowed on kind='directive' — an
unreviewed standing rule is a contradiction in terms; use
provenance='agent' (or have a human endorse it) instead

That rejection is a backstop, and in practice a second constraint does most of the work. The one mechanism that actually writes auto notes in my setup is a git hook that captures what a commit touched. It writes them at the lowest priority under the kind reserved for ordinary learnings, and that kind has no unsolicited delivery path at all. An auto note therefore cannot reach a session unless the session went looking for it. Two independent reasons an unreviewed capture never crowds out a rule you set: one enforced at the schema, one falling out of how the writer was configured. I would not want to rely on either alone.

What provenance is not. It is a caller-declared field, not a verified identity claim. Nothing cryptographically proves that the process which wrote provenance="agent" was an agent. What the design buys is narrower and still worth having: an honest caller cannot accidentally overstate its authority, and the one class that carries real weight requires an action on a surface a person operates. If your threat model includes a hostile writer with direct store access, you need signed writes, and that is a different post.

Interactive version: the post on my site has a demo where you change the class and the kind and watch the identical sentence arrive with a completely different weight, including the combination that is refused outright.

4. Promotion: Trust Is Earned, Never Asserted

A ladder you can only be placed on is not much use. Notes get reviewed. An auto-captured note about a commit turns out to be exactly the context a later session needed, and a session verifies it against the code. That review is real information and the store should keep it.

So there is a promotion operation, with three rules that matter more than the operation itself.

One step at a time. auto to agent, or agent to human. Never auto straight to human. The implementation computes the note's current rank and rejects any target that is not exactly one above it. Skipping a rung would mean a single review had substituted for two independent ones.

No demotion. There is no operation that lowers a note's class. If a note turns out to be wrong, that is not a trust-level problem, it is a revocation, which is the next section. Demotion would let a bad actor or a confused session quietly strip authority from a rule the user set, and the recovery path for that is worse than the problem it solves.

The top rung is not the agent's to hand out. The promotion tool exposed to agents has its target parameter constrained to a single value: agent. Its own description says so plainly, in the text the model reads when deciding whether to call it:

Raise an auto-captured note's trust class to 'agent' — e.g. after this
session has reviewed an auto-captured note and confirmed it still holds.
This tool only takes that one step (auto -> agent); it never promotes a
note to 'human', because deciding that a person has endorsed something is
not the agent's call to make. Human endorsement happens on a user-side
surface instead (a CLI/UI a person operates), not through this tool.

The underlying store function does support the agent to human step. It has to, or human endorsement would be impossible. What differs is which surface can reach it. The tool layer an agent talks to caps out one rung early, and the final step happens somewhere a person is actually present. Two layers, two different ceilings, one boundary that a prompt cannot argue its way across.

The generalizable rule. Any trust system where the subject can self-certify is not a trust system, it is a formality. If your memory store lets the agent write its own authority level, delete the field. It is costing you schema space and giving you nothing. The value comes entirely from the class that requires an act the agent cannot perform.

An honest note on how much this gets used: promotion is the least-exercised operation in the system. Reviewing old auto-captured notes is nobody's idea of a good time, and mostly it happens as a side effect, when a session pulls up an auto note for some other reason, confirms it against the code, and promotes it on the way past. I would not build the promotion path first. I would build the framing first, notice that some notes deserve better, and add promotion when that starts to annoy you.

Both promotions and the lifecycle events below are appended to a per-note event log, with an actor recorded on each. A promotion to human logs actor human, and a promotion to agent logs actor agent. The current class is a fold over that log rather than a value someone overwrote, which means a note's trust history is auditable after the fact: you can see that it started as an auto-capture, was reviewed in July, and endorsed in August.

5. Revocation Is Not Deletion

This is the design decision I argue about most, and the one I am most confident in.

When you discover that a stored note is wrong, the intuitive move is to delete it. Bad data, remove it, done. I did that first. Three weeks later the same wrong belief was back in the store, written by a different session, in almost identical words.

Obvious in hindsight. The note was wrong because a reasonable agent, reading that code, drew a reasonable and incorrect conclusion. Deleting the note removes the conclusion but not the code that invites it. The next session walks the same path, makes the same inference, stores the same mistake. Deletion is not a fix, it is a reset button on a loop.

Why the tombstone is the vaccine. A deleted wrong note leaves the store in exactly the state it was in before the mistake was ever made, which is the state in which the mistake gets made. A revoked note leaves it in a strictly better state: the trap is still there, and now there is a sign in front of it. The sign is worth more than the empty space.

So revocation appends an event rather than removing a row. The note stays a live candidate for recall and for injection. What changes is what gets rendered in its place. Instead of the note's content, every surface substitutes a fixed template:

Previously believed (recorded {created_date}, revoked {revoked_date},
reason: {reason}): "{summary}". Do not re-derive this from other sources
without verification.

Read the last sentence again, because it is the load-bearing one. The template does not merely announce that something was wrong. It gives an instruction aimed at the specific failure: you are about to reason your way back to this belief from the same evidence, and you should not, without checking. The tombstone is addressed to the future session that is about to repeat the mistake.

The retrieval property that makes it work

Appending an event leaves the note's embedding untouched, and that embedding was computed from the original wrong content when the note was written. It sounds like an implementation shortcut. It is the most useful consequence of the whole design.

Because the vector still points at the wrong belief, the tombstone comes back for exactly the query that would have surfaced the mistake. A session working on locking, asking about lock scope, gets the struck-through page. A session working on billing never sees it and pays nothing for its existence. The deterrent rides the same similarity machinery that delivered the error, which means it is aimed at the moment of maximum risk without anyone having to aim it.

The correction and the revocation are one write

When a session discovers the truth, it usually wants to record the right fact and mark the old one wrong. Doing that as two calls means there is a window where the store contains both the wrong note and the right one, un-linked, and if the second call fails you are left with a contradiction and no arbiter. So the write call takes a contradicts parameter: record this new note, and in the same transaction, append a revoked event to the note it corrects, with the reason pointing at the new note's id. One write, no window, and the tombstone names its replacement.

Revocation does not set a validity window

There is a separate mechanism for supersession, where a note is closed out with a valid_until timestamp because a newer note replaced it. Revocation deliberately does not touch that field. A superseded note is old news and can fall out of the working set. A revoked note is wrong, and it needs to stay in front of readers precisely because its content is attractive. Those are different lifecycle facts and conflating them would make the wrong note quietly disappear, which is the behavior I was trying to eliminate.

Revocations are themselves sometimes wrong

A session decides an old note is wrong, revokes it, and is itself mistaken. This happens. It happened to me while I was building the feature, which is how it got built. So reinstatement exists, and it is always legal: append a reinstated event, the fold takes the latest transition, and the note's original content comes back. No special case, no permission check about who revoked it first, and no limit on how many times a note can flip.

That last point sounds sloppy and is not. The alternative is arbitration logic that decides which of two disagreeing sessions was right, and no such logic can be correct, because the store has no independent access to the truth. Appending both transitions and letting the fold report the latest keeps the full history queryable and keeps the code honest about what it does not know.

One guard rail sits under all of this: the actor on a revocation can never be the system. A staleness flag is machine-derived and gets recorded as such, but calling a note wrong is a judgment, and the store refuses to accept a revocation that claims otherwise. Nothing in the pipeline revokes anything on its own.

Deletion still exists, and it is for a different job. There is still a plain delete. Its job is notes that are irrelevant, not notes that are wrong: a task note for finished work, a duplicate, something written by mistake in the wrong workspace. Confusing the two is the common error. If a future reader would benefit from knowing you once believed the thing, revoke. If knowing that would be pure noise, delete.

Here is what one note looks like at each point in its life. Active:

[47] [HIGH] [GOTCHA] [agent] [scope=repo]  [locking, resolver]  (7w ago)
  Memory to verify (recorded by an AI session, not human-endorsed):
  lock_workspace() at resolver.rs:214 acquires a PID-scoped lock; it drops
  when the function returns.

Revoked:

[47] [REVOKED]  [locking, resolver]
  Previously believed (recorded 2026-06-14, revoked 2026-08-02, reason:
  lock is released on scope exit, not on return): "lock_workspace() drops
  its lock on return". Do not re-derive this from other sources without
  verification.

And the event log behind it, which is what state is folded from:

created      actor=agent  2026-06-14
revoked      actor=agent  2026-08-02  reason="lock is released on scope exit, not on return"
reinstated   actor=agent  2026-08-02
revoked      actor=agent  2026-08-02  reason="contradicted by #48"
folded state: revoked      in recall results: yes

Interactive version: the post on my site lets you drive this one yourself.

6. Staleness: A Changed Anchor Means May, Never Is

The second failure mode, right then stale, is the one with actual observable evidence. A note that says something about agent/searcher.py can be tied to that file, and the file can be checked.

The mechanism is unremarkable. A note can declare anchors: file paths, each stored alongside a truncated SHA-256 of the file's content at write time. On recall, the current content is hashed with the same function and compared. Mismatch means drift.

Three other signals feed the same flag, and they overlap deliberately. A referenced file whose modification time is later than the note's creation time. A stored hash of the specific code block the note described, if the note captured one. And explicit supersession, where a newer note has closed this one out. Any of these fires the flag.

What matters is what happens next. The drifted note is not dropped, not down-ranked, and not marked false. It gets extra lines:

[47] [HIGH] [GOTCHA] [agent] [scope=repo]  [locking, resolver]  (23d ago) [STALE]
  Memory to verify (recorded by an AI session, not human-endorsed): lock_workspace()
  at resolver.rs:214 acquires a PID-scoped lock; it drops on scope exit.
  WARNING: These files changed after this note was written: agent/resolver.rs [anchor_changed]
  VERDICT: anchor changed since — verify: agent/resolver.rs [anchor_changed]
  WARNING: Verify this note is still accurate before relying on it.

The verdict line names the file and says verify. It does not say the note is wrong, because the check does not know that. This distinction is not pedantry, it is the difference between a useful signal and a boy-who-cried-wolf detector.

Anchor drift as a classifier. Treat drift detection as a binary classifier for the event "this note is now false." Write D for drift detected and F for the note actually being false. What we have is a detector with very high recall and poor precision:

P(D | F) ≈ 1 and P(F | D) ≪ 1

The first says almost nothing becomes false without the anchored file changing, so drift catches nearly every genuine staleness case. The second says most drift is harmless: the file changed for unrelated reasons. Recall near 1.0, precision low, which is the profile of a screening test.

You handle a screening test by routing positives to a confirmatory step, never by treating a positive as a diagnosis. The confirmatory step here is the agent re-reading the file, which the verdict line asks it to do. Down-ranking or hiding drifted notes would be treating a screen as a diagnosis, and would throw away correct notes at the low precision rate.

The high-recall half comes with a condition attached, and that condition is where the mechanism is weakest: the note's truth has to be determined by the content of the files it anchors to. A note about a function in resolver.rs qualifies. A note about how two services interact at runtime does not, and neither does a note nobody anchored to anything, which is most notes in most stores. For those, P(D | F) is not near 1. It is near 0, and drift detection is simply not a control that covers them. Elapsed time is the only signal left, which is why process and environment notes carry a last-confirmed date on top of the hash check.

One case gets special handling, and I think it is the most interesting part of the staleness design. Some notes describe processes rather than code: how the build works, which flags CI passes, what the environment needs. There is no single source file that is that fact. What you can anchor to is a proxy: the lockfile, the CI config, the Dockerfile. That anchor stands in for the process it encodes, not for the note's content directly.

So when a proxy anchor drifts, the honest reading is weaker than for a code anchor. It means the process this note describes may have changed. The rendering says exactly that, appending to the standard verdict rather than replacing it: the verify instruction stays, and a clause is added naming the proxy and the date the fact was last confirmed. Notes of this kind also carry a last-confirmed date even when nothing has drifted at all, because environment facts decay by elapsed time rather than by hash mismatch. A six-month-old claim about a CI runner is suspect whether or not the config file changed.

The framing rule I ended up with. Every status line in the recall output comes from deterministic machine state: a hash matched or it did not, a date is what it is. No adjectives, no model-produced confidence, no "this note seems outdated." If the system cannot compute it, the system does not claim it. That constraint killed several features I wanted and made the remaining ones trustworthy.

7. Delivery Rules: Trust Decides Where a Note Shows Up

Provenance answers how much to trust a note. A second stored field, confusingly named kind, answers something different: when should this note appear without anyone asking for it? Every note declares one, from a short list, and the choice determines whether the note waits to be searched for or shows up on its own.

Unsolicited delivery is expensive. It spends context window on every session whether the note turns out to be relevant or not, and a store full of standing rules is a store whose rules get skimmed. So the right to interrupt is rationed by kind, and the kinds with the most interrupting power are the ones provenance restricts.

Kind	When it arrives	Why that rule
directive	Every session start, and again after compaction, at any priority	A standing rule that is missed once is a rule that does not exist. This kind pays the unconditional cost, which is why an unreviewed capture is not allowed to hold it.
task	Session start, high priority only	Resuming work needs the current state, but only the state that actually matters. Medium and low priority task notes stay recall-only.
gotcha	When one of its anchored files is about to be edited	A caveat about a file is worthless three days early and priceless three seconds before the edit. The anchor is both the staleness check and the delivery trigger.
finding	Relevance-ranked, on the prompt or on an explicit recall	The default. Learnings are numerous and situational; they compete on similarity rather than arriving unconditionally.
decision	On demand only, recallable in chronological order	An architectural decision is meant to be read as a sequence with its neighbours, not pushed at a session that did not ask.

The trap everyone hits in week one. A gotcha is delivered by its anchors. Write one with no anchor and it gets an empty delivery bundle: no path to fire on, so it never fires, and it looks from the outside exactly like a broken feature. It is not broken, it is a caveat about nothing in particular. The store could reject anchorless gotchas outright and probably should, but they are still perfectly good recall targets, so it accepts them and they sit there quietly. If you build this, put the warning at write time.

Two more constraints keep the pushed bundle from becoming its own problem. It is budgeted, so a store with two hundred notes does not hand a session two hundred notes, and a note that does not fit is dropped whole rather than truncated mid-sentence into something that reads like a different claim. It is deduplicated by note id within a turn, so a note that qualifies through two triggers at once arrives once.

And the wrapper text around the whole bundle scales to the weakest note inside it. If everything in a batch is human-endorsed, the envelope says so. Slip one auto-captured note into the same batch and the envelope drops to the weaker wording, because the reading agent is about to consume the batch as one blob and calibrating it to the strongest member would be a lie about the rest.

None of this is free, and the bill lands in the one place you cannot expand. The auto frame runs about a hundred characters, call it twenty-five tokens, prepended to every note that carries it. A twenty-note recall pays that twenty times, so you can spend five hundred tokens of a session's context on framing alone before a single fact arrives. That is not nothing, and it is the reason the frames are terse and fixed rather than generated per note. If your bundles run large, the honest move is to frame the envelope once and mark individual notes with a short tag, which trades per-note clarity for headroom. I have not needed to. Ask me again when someone runs this on a store with ten thousand notes.

The pushed path also uses a different framing template from the one in section 3. When a session explicitly asks for a note, it has already decided to weigh it, so the provenance-hedged wording applies. When the system pushes a note at a session that did not ask, the framing states structural facts instead: the date it was recorded, the anchor it is tied to, and the anchor's current status.

Recorded {date} (anchor: {target}, status: {status}): {content}

status ∈ { "matches current state",
           "changed since — verify",
           "last confirmed {date}" }        # process and environment facts

Two framings for two situations. A pulled note is answering a question the agent asked and the useful signal is who stands behind it. A pushed note is interrupting, and the useful signal is whether the world it describes still looks the way it did. Same trust model, different projection of it.

Interactive version: the post on my site has a demo where you pick a moment in a session and see which of five stored notes arrive unasked, and why.

Part 3: What I Have Not Solved

8. The Missing Middle Class

Three classes are not enough. I know where the fourth goes and I have not built it.

Picture a normal Tuesday. You type, in your own words, in a chat turn: never run the benchmark suite against the global Python install, always use the project virtual environment. The agent does exactly what it should and records that as a note, so the rule survives into future sessions. The note goes in through the agent-facing memory call, which means it gets provenance="agent", because that is what the enum allows.

Now look at what a future session receives:

Memory to verify (recorded by an AI session, not human-endorsed):
never run the benchmark suite against the global Python install,
always use the project virtual environment.

Every word of that framing is technically accurate. An AI session did record it, and no human endorsement step took place. And it materially understates the authority of the content, which is a direct transcription of something you said. The framing tells the reading agent to verify a rule that was never in question.

Now the mirror failure, which is worse. An agent half-remembers a conversation from ninety turns ago, decides the user probably wanted commits squashed before merge, and records it as a finding. Same class, same framing, no banner distinguishing it from the transcription case. In the first case the framing under-sells the truth. In the second it over-sells a fabrication. The class is doing the same thing in both, which means it is doing nothing.

The shape of the gap. The ladder currently measures who performed the write. What it needs to measure is whose claim this is. Those come apart exactly when an agent transcribes a user, which is one of the most common and most important writes a memory system ever handles.

The obvious fix is a fourth class between agent and human: user-stated, agent-transcribed. Something like "the user said this; an AI session wrote it down." That gives the reading agent the right posture, which is: treat this as authoritative, and if it looks wrong, ask rather than silently discount it.

A new class alone fixes only half of the problem, though, and the half matters. It repairs the transcription case, where a true user rule was being hedged. It does nothing for the mirror case, because an agent that misremembers a conversation would reach for the new class just as readily as it reached for agent, and would then be believed. The class is only worth adding if something other than the agent's own say-so decides who goes in it. Working out what that something is turned out to be harder than the class, which is the next section.

9. Mixed Provenance, and the Attestation You Cannot Delegate

Real notes are not pure. This one is from my own store, lightly edited:

User: never add query-side keyword heuristics to the ranker. This is the
third time this rule has come up. It applies to the searcher and to the
reranker, and the reason is that keyword branches cannot generalize across
languages; the two index-time priors handle the cases the heuristics were
covering.

The first sentence is the user's rule, close to verbatim. The second is a fact about the conversation. The rest is the agent's own synthesis: where the rule applies, why it exists, what replaced the deleted code. Some of that synthesis is correct. Some of it is the agent's interpretation, and interpretation is exactly where the wrong-when-written failure lives.

One class per note cannot represent this. Stamp it agent and the user's rule gets hedged. Stamp it at a user-stated class and the agent's interpretation inherits authority it did not earn. Both choices are wrong, and splitting the note into two is not a real answer either: the rule and the reason belong together, and a memory system that forces users to write in provenance-pure fragments will not be used.

The candidate design

What I keep coming back to is evidence rather than assertion. The harness hook that fires when you submit a prompt already sees your raw text before the model does. That text is the ground truth, and it exists outside the agent's reasoning. So: bind the verbatim excerpt to the note as an evidence field, and at render time check each span of the note against it. Spans that match the recorded user turn render as user-stated. Everything else renders at the agent class.

note.content  = "User: never add query-side keyword heuristics to the ranker.
                 It applies to the searcher and to the reranker, and the
                 reason is that keyword branches cannot generalize."
note.evidence = { turn_id: 4417,
                  excerpt: "never add query-side keyword heuristics" }

render:
  [user-stated]  never add query-side keyword heuristics
  [agent]        to the ranker. It applies to the searcher and to the
                 reranker, and the reason is that keyword branches
                 cannot generalize.

The property that makes this worth building is that the check is machine-verifiable. The system is not asking the agent whether the user said something, it is comparing a stored string against a recorded turn. An agent can still choose a misleading excerpt or bound it badly, but it cannot manufacture text the user never typed.

The attestation you cannot delegate. The tempting shortcut is a boolean on the write call: user_stated=true. That is the agent certifying its own authority in a different costume, and it fails the same way. "The user said this" is exactly the claim that must be checkable against something outside the agent's control loop, because it is the claim with the most to gain from being wrong.

The unsolved parts, honestly stated. Span boundaries are fuzzy: paraphrase, translation, and the user's own typos all break exact matching, and fuzzy matching reintroduces a judgment call. Rendering gets noisy fast if every note arrives striped with class markers. Storing verbatim user turns alongside notes is a privacy surface that needs its own retention rules. And there is a real question of whether the payoff justifies it, which is the subject of the last section, because I have not measured whether any of this framing changes behavior at all.

10. The Banner Is Unmeasured

Everything above rests on an assumption I have never tested: that the framing changes what the reading agent does.

It is a plausible assumption. Prefixes shift model behavior in general, and this one is unusually direct. It is also exactly the kind of plausible assumption that turns out to be false, and I have not run the experiment. Neither, as far as I can find, has anyone else. The academic work on memory provenance mostly evaluates whether the right memory was retrieved, not whether the trust framing on it changed the downstream action.

The failure mode I worry about is not that the banner is ignored. It is that the banner works too well in the wrong direction: an agent that reads "memory to verify" on a correct standing rule and, being agreeable, verifies instead of complying. Now every correct rule costs an extra tool call and a chance of the agent deciding the rule does not apply. That is a real regression, paid on every note, in exchange for protection against notes that are wrong some fraction of the time.

The experiment that would settle it

Two arms, banner on and banner off, over two scenario families.

Family A, the correctly recorded directive. Put a true standing rule in the store, one whose violation is unambiguous and detectable in the transcript. Run the task. Measure the compliance rate in each arm. The difference is the cost of hedging a true rule. Call it L.

Family B, the deliberately misrecorded directive. Put a note in the store that is confidently wrong in a way the agent could catch by checking the code. Run a task where acting on the note causes visible damage. Measure how often the agent verifies before acting. The difference between arms is the protection the banner buys. Call it G.

What the banner is worth. Pick a unit you actually care about. Minutes of engineer time is the one I use, because both sides of this trade end up costing somebody minutes. Let p be the fraction of recalled notes that are actually wrong, G the minutes saved each time the banner makes the agent catch a wrong note before acting on it, and L the minutes burned each time the banner makes the agent second-guess a note that was fine. The expected value per recalled note is:

V = p · G − (1 − p) · L

The banner pays for itself when V > 0, which is when the wrong-note rate exceeds a break-even threshold:

p* = L / (G + L)

Only the ratio between G and L matters for the threshold, so the choice of unit cancels out and you are left arguing about one number: how many times more expensive is acting on a wrong note than double-checking a right one? My working guess is twenty to one. Chasing a bad assumption through a debugging session costs the better part of an hour; an extra file read costs a couple of minutes. At that ratio the banner pays on any store where more than about five percent of notes are wrong, which every store I have looked at clears easily. That is a guess dressed in arithmetic, not a measurement, and if L is really closer to G because a hedge derails compliance outright, the whole design is upside down.

The measurement has to be per class rather than aggregate. The auto framing is much stronger language than the agent framing, and pooling them would average away the effect you are trying to see. The other thing that would quietly ruin the experiment is making the family B notes too obviously wrong. A note claiming the sky is green tests nothing, because any competent model catches it banner or no banner. The note has to be wrong the way the lock note was wrong: plausible, specific, and only detectable if you go and look.

Until that runs, the honest status of the trust ladder is that it is coherent, cheap, and unfalsified. Which is not the same as known to work, and I would rather say so than let the tidiness of the design stand in for evidence.

Interactive version: the post on my site has sliders for p, G and L so you can watch the break-even point move.

11. What I Would Build Next

The lock note from the first paragraph is still in my store. It carries a tombstone now: previously believed, revoked, reason, do not re-derive this without verification. I have watched a later session hit that tombstone while reading the same code, pause, and check. That is the single most convincing thing I have seen come out of this work, and it is an anecdote, not a result.

If you are building a memory layer, the cheapest useful thing you can do this week is add a provenance field with a closed enum and render it on recall. A column and a string concatenation. What costs you is everything that follows once you take it seriously, and the hardest of those is not a feature at all: it is holding the line that the system never claims more than it can compute. Every time I relaxed that, I got a nicer-sounding recall block and a worse one.

Two things go on my list next. Binding verbatim user turns to notes as evidence, because the transcription case is common and currently fails in both directions at once. Then the banner experiment, because I have built a fairly elaborate structure on top of an untested premise and I would like to know whether the premise holds. If it does not, most of part 2 of this post is scaffolding around a no-op, and I would rather find that out from an experiment than from a reader.

The through-line is easy to state and hard to hold to. A memory system that only stores and retrieves smuggles a claim into every response: this is true. Nothing in its architecture can support that claim. The way out is not storing less, it is being exact about what each note is, who stands behind it, whether the ground under it has moved, and whether anybody has since found it wrong. On anything that runs longer than one session, that precision is what makes the memory usable rather than merely present.

The Sentinel Pattern: A Multi-Agent Dev Loop That Doesn't Trust Its Own Agents

Swapnanil Saha — Thu, 06 Aug 2026 21:44:22 +0000

A coder agent finished its lane, wrote up a tidy report, and told me it had implemented nine of the things I asked for. I wrote nine into the ledger and moved on. Later, doing something unrelated, I enumerated the artifacts myself and counted seven.

Nothing dramatic followed. Nobody shipped a bug; I caught it in time. But by then the number was already in the file that the rest of the loop reads from, which meant a later task had cited it, which meant a wrong fact had acquired provenance. That is the failure I keep coming back to, because it is not a model failure and it is not fixable by better prompting. The agent produced its report from its own compressed working context, and compression drops exactly the details you would want to audit. It wasn't lying. It was summarising, and summaries round.

This post is the development process I actually use to build vectr, a semantic code search and working-memory tool, from one MacBook and one subscription. One orchestrating agent that writes no product code, a rotating cast of disposable specialists, and a gate in the middle that treats every claim as hearsay until an artifact backs it. The roles come first because they are the easy part. Then the rules, which are the hard part, because every one of them was written by something going wrong.

Part 01: The Shape

01. The Unit of Work Is Not the Agent

Start with the constraint, because the constraint is what makes this interesting. I have one machine, one Claude subscription, and a task list longer than my attention span. Working with a single agent, my throughput is bounded by how fast I can read what it did. That bound is real and it is low. Reading a diff carefully is slower than generating it, and it has been slower ever since the models got good.

So you reach for parallelism. Five agents, five tasks, five branches. I call one of those a lane: one agent, one scoped task, one branch, running start to finish without supervision, and the word will do a lot of work in this post. The first version of that setup works beautifully right up until you try to merge, at which point you discover the thing nobody puts in the architecture diagrams: parallelism multiplies output and unverified claims at exactly the same rate. Five lanes produce five diffs and five reports. The diffs are real. The reports are a summary of the diffs written by the same process that wrote them, from a context that has already been compressed once or twice along the way.

diff: the set of changes between two versions of the code, shown line by line. It is what the agent actually did, as opposed to what it says it did.

"The model hallucinates" is the wrong diagnosis here, and it sends you off tuning prompts when the problem is structural. A subagent working a long task fills its context window, gets compacted, keeps going, gets compacted again. By the time it writes "I implemented nine", the nine is not a count. It is a recollection of a count, reconstructed from a summary of a summary. Ask a human to write a status report from memory after eight hours of work and you get the same class of error, which is why we invented commit logs.

subagent: a subordinate agent launched by another agent to do one scoped task, with its own separate conversation and context window.

context window: the fixed number of tokens a model can attend to at once. Everything the agent knows in the moment lives inside it; anything that falls out is gone unless it was written down.

compaction: when a long conversation is summarised to reclaim context space. The summary keeps the gist and loses exact counts, signatures, and line numbers.

The coupling you are trying to break

Agents are cheap and verification is not. If your process scales work and verification together, you have not parallelised anything; you have moved the queue from generation to review and made the queue longer. The whole point of a gate is to make verification cost less than generation, so the two can scale at different rates.

The pattern that fell out of this has one long-lived agent and many short-lived ones. I call the long-lived one the sentinel. It writes no product code. Its entire job is to scope work, launch specialists, attack their output, and decide what gets merged. Everything else in this post is a consequence of that split, and of the fact that I got the split wrong several times before it stuck.

02. Three Roles, and What Each One Is Trusted For

Three kinds of agent run in this loop. What separates them is not capability, since they are all instances of the same family of model. What separates them is what I am willing to believe from each one without checking.

The sentinel

Orchestrator and gatekeeper. It writes the briefs, launches the lanes, audits the results, owns the ledger, and merges. Two rules define it, and both are load-bearing.

First, it writes no product code. The practical reason is context: auditing four lanes is the most context-hungry job in the system, and an orchestrator that has also been elbow-deep in a refactor has spent that context. The structural reason is that a reviewer with a diff in the merge has an interest in the merge succeeding. I do not think a language model experiences that as a conflict of interest the way a person would, but I would rather not find out at the gate.

Second, anything paid or irreversible runs from the sentinel. Pushes, tags, releases, and evaluation runs that consume the shared quota window. Not because the specialists would be reckless with them, but because those actions have no undo and I want exactly one place where they can originate. A lane can propose a release. It cannot cut one.

The coders

One well-scoped task each, one git worktree branch each, the full test suite green before reporting, and a structured report back in a shape I specify in the brief. When the lane ends, the agent ends. There is no long-lived coder that accumulates knowledge across tasks, and that is deliberate: accumulated context is unaudited state, and unaudited state is the thing this entire process exists to minimise.

git worktree: a second working directory checked out from the same repository, on its own branch, sharing one object store. Two worktrees cannot overwrite each other's files.

The obvious objection to disposable coders is that every new one has to re-learn the codebase, at full token price, from scratch. That objection is correct and the cost is real. It is the single biggest inefficiency in the whole loop, and the only thing that makes it tolerable is that findings get written down where the next lane can retrieve them instead of rediscovering them (section 11). Without that, disposable agents would be indefensible.

Coders get tight briefs. Task, the files they may touch, the rails they must not cross, and the exact shape of the report. A vague brief on a mid-tier model produces a lane that solves an adjacent problem beautifully.

Two details in the report shape do more for me than everything else in the brief combined. Ask for raw command output, pasted, rather than the lane's account of what the command said, because the account is where the rounding happens. And ask for the commit SHA the report describes. Without a pinned SHA the audit races the lane: it can push another commit while I am reading, and then I have verified a state that no longer exists.

TASK    one sentence, one outcome, no "and also"
WHERE   worktree wt-<name>, branch feat-<name>
TOUCH   src/searcher/*.py, tests/test_searcher.py
RAILS   do not touch the index schema; no destructive git;
        no new dependencies; no changes outside TOUCH
DONE    full suite green in your worktree
REPORT  1. commit SHA
        2. `git diff --name-only main...HEAD`, output pasted verbatim
        3. suite summary line, pasted verbatim (passed/failed/skipped/collected)
        4. anything you tried that did not work, one line each
        5. write your key findings to shared memory before you finish

That fits on a screen and takes ten minutes to write, most of it spent on the TOUCH and RAILS lines. Those two are what stop a lane from helpfully refactoring something three modules away, which is the failure mode that turns a clean merge into an afternoon.

The adversarial reviewers

Separate agents whose only job is to attack the work, each on an independent axis. In my loop those axes are: does retrieval actually return the right thing, does the tool get adopted by the agent it is built for or ignored, and does it beat a no-tool baseline on turns and tokens and wall time. Three different questions, three different agents, deliberately not one reviewer asked to consider all three, because a single reviewer with three mandates will find the easiest failure on the easiest axis and stop.

The instruction that made reviews useful was giving them the branch and a change note rather than a diff. A reviewer handed a diff reviews the diff. A reviewer handed a working build and told to go break it will use the product, wander outside the diff, and find the thing the diff broke two modules away. That is where the real defects have come from.

Two guardrails on reviewers. Rounds are capped, because an adversary with unlimited rounds will always find another nit and you will never ship. And reviewer findings are independently verified before they gate anything, because reviewers invent defects at roughly the same rate coders invent completions. A fabricated defect that blocks a merge costs a full extra loop, which is the most expensive kind of wrong.

Role	Writes product code	Lifetime	Trusted for	Never trusted for
Sentinel	Never	Long, spans the sprint	Scope, merge decisions, the ledger, paid and irreversible actions	Nothing it did not verify itself
Coder	Yes, in its own worktree	One task, then discarded	Producing a diff and a green suite in its lane	Counts, completion claims, cross-lane impact
Reviewer	Never	One capped review round	Pointing at where to look on its axis	The finding itself, until reproduced

The kitchen, not the assembly line

Most multi-agent diagrams look like an assembly line: work moves right, each station adds something, the last station ships. A working kitchen is closer. Cooks work in parallel on separate stations, and everything passes the expediter at the pass, who cooks nothing, tastes everything, and sends plates back. The expediter is not a stage in the line; they are the reason the line can run in parallel at all, because they are the single point where "done" gets decided. When a kitchen gets slammed, the pass is where it backs up, and that is the design working rather than failing.

03. The Ledger: One Append-Only File

Every task, every decision, every merge, and every gate verdict goes into one file, appended, never rewritten. It sounds bureaucratic for a solo project. It is the most valuable artifact in the loop.

The reason is that conversations do not survive. A chat window gets compacted, and when it does, the exact number in message forty gets replaced with a paraphrase. Sessions end. Machines sleep. Agents die mid-sentence. The ledger is the only thing in the system with a memory that does not degrade, so it is the only thing allowed to be authoritative about what happened.

Append-only matters more than it looks. A mutable status file lets a wrong entry be quietly corrected, which sounds like a feature until you realise that a silently corrected error teaches you nothing. When the count of nine turned out to be seven, the correction went in as a new line underneath, next to the original claim and the enumeration that settled it. That line is why the rule exists at all. A file that hid the error would have produced a small tidy edit and no rule.

[2026-07-14] TASK  retrieval-rerank-blend
  brief:  blend owning-class prior into rerank; touch searcher + index only
  lane:   coder / worktree wt-rerank / branch feat-rerank-blend
  report: "8 call sites updated, suite green (412 passed)"
  AUDIT   grep -c over the branch diff: 8 call sites  [matches]
  AUDIT   suite re-run on MAIN checkout: 412 passed, 0 skipped  [matches]
  verdict: MERGE  c1f9a2e

[2026-07-14] CORRECTION to entry [2026-07-11] docs-sweep
  ledger recorded 9 updated files on the lane's report
  enumeration of the branch diff: 7 files
  root cause: report written post-compaction; count was recalled, not counted
  rule added: no merge on a lane's claim, enumerate the artifacts

Two things that entry does. It records the evidence, not just the verdict, so a later reader can tell whether the merge was justified or lucky. And it makes the audit a written step rather than a thing I intended to do. If the AUDIT lines are missing, the merge did not happen, because I cannot tell afterwards whether I checked.

The ledger doubles as the recovery artifact. If I lose a session entirely, the ledger plus git branch --list reconstructs the state of the world in about a minute: what was assigned, what merged, what is still open on a branch somewhere. That property has saved me more than once, and it is worth designing for on purpose rather than discovering afterwards.

Part 02: The Gate

04. Rule 1: Never Merge on a Claim, Enumerate the Artifacts

A report is hearsay, an artifact is evidence, and the gate only accepts evidence. Everything below is what that costs in practice.

Every claim in a lane's report has to be convertible into something I can produce myself in under a minute. "I updated nine files" becomes a file list from the branch diff, and I count the list. "The suite passes" becomes me running the suite, on the main checkout, and reading the summary line including the skip count. "I added the regression test" becomes finding the test by name and reading its body to check that it would actually fail without the fix. If a claim cannot be turned into an enumeration, I rewrite the brief so that next time it can, because a claim that cannot be checked is a claim that will eventually be wrong in a way nobody notices.

This is not paranoia about models. It is a cost asymmetry. Enumerating a claim costs a minute or two. A wrong claim in the ledger costs however long it takes to notice, plus everything that was built on top of it in the meantime, plus the time to work out which downstream conclusions are now suspect. When one side of a trade costs minutes and the other costs hours with a long tail, you do not need a strong prior about failure rates to know which way to lean.

The second-order failure: laundering

The reason a wrong count is worse than it looks is that the ledger confers authority. Once nine is written down, the next lane reads nine from the ledger and repeats it in its own report, and now the error has a citation. It has stopped being one agent's summary and become a fact of the project with a paper trail. Errors laundered through a trusted record are very hard to see, precisely because everything downstream is consistent with them. The only defence is that the record itself has to be built from artifacts.

One nuance worth stating, because it changes how you write briefs. The enumeration should be something the sentinel does, not something the lane does and reports. Asking a coder to "verify and confirm the count" produces a confirmation, which is another claim from the same source. Verification has to cross an agent boundary or it is not verification. This is the same reason we do not let people mark their own exams, and it is a much older idea than any of this tooling.

Turning a claim into evidence

The useful skill at the gate is spotting the gap between what a sentence asserts and what it actually proves. Three claim types, and the audit each one needs.

A count. The report says: "Updated 9 files across the docs sweep. Lane complete." What that does not prove: nothing about the number. The report was written after the lane had been compacted twice, so 9 is a recollection of a count. The sentence is equally fluent whether the true answer is 9, 7, or 12. The enumeration: ask git, not the agent. git diff --name-only main...feat-branch prints one path per line; count the lines yourself. What came back: 7 paths. Two of the nine were files the lane opened, considered, and left unchanged. Both appeared in its narration, neither appeared in the diff. The gap is two files that were read rather than written, which is exactly the distinction a summary flattens.

A green suite. The report says: "Full test suite passes." What that does not prove: which suite, on which checkout, with how many tests skipped. A run inside a dot-directory worktree can quietly skip path-walking tests and still print a green line. "Passes" hides the denominator. The enumeration: run it yourself on the main checkout and read the summary line verbatim: passed, failed, skipped, collected. Compare the collected total to the last known-good run. Settled when the collected count matches or exceeds the previous run, skips are zero or individually explained, and the run happened where the code will actually live. The interesting number in a test report is never the failures; it is the collected total, because a suite that silently collects fewer tests than yesterday is green for the wrong reason and will stay green all the way to production.

A regression test. The report says: "Added a regression test for the fix." What that does not prove: that the test exercises the bug. A test can be present, named correctly, pass, and assert on something adjacent to the actual defect. It then passes forever, including on the broken code it was written to catch. The enumeration: find the test by name, read its body, then invert it: revert the fix and confirm the test fails. A regression test that cannot fail is documentation with a green tick. This is the claim I audit hardest, because a fake regression test does not merely fail to catch the bug once. It certifies that the bug cannot come back, and everything downstream trusts that certificate.

The count case is a real event from this project. The other two are the same procedure applied to the two claim types I audit most often; they show the method and the standard of proof rather than a recorded outcome.

05. What the Gate Costs: the Arithmetic of a Single Bottleneck

Now the part that decides how many agents you should actually run, which is a question I have never seen answered with a number.

Lanes run concurrently. Audits do not. There is one gate, it is me plus the sentinel, and it processes one lane's evidence at a time. That single fact is enough to derive the ceiling on the whole approach.

Numbers first, algebra after. Say a lane takes 40 minutes and auditing it takes 8. Done serially, four tasks cost 4 × 48 = 192 minutes. Run the four lanes at once and they all finish at minute 40, then the four audits go back to back and the last verdict lands at minute 72. That is 2.67 times faster, not 4 times, and the gap is entirely the audit queue.

Generalising: let L be lane duration, A be audit duration, N the number of concurrent lanes.

Speedup with a serialised gate

For one batch of N identical lanes that start together:
T_serial   = N (L + A)
T_parallel = L + N A

S(N) = N (L + A) / (L + N A)
As N grows the N A term dominates the denominator, so the speedup converges:
lim S(N) as N → ∞  =  (L + A) / A  =  L/A + 1
Your maximum useful fleet size is set by the ratio of lane work to audit work, and by nothing else. Lanes of 40 minutes with 8-minute audits cap at 6 times. Not 6 times if you get the tooling right: 6 times, full stop, and the tenth lane costs ten lanes of tokens to buy a rounding error.

That formula describes one batch. A project is not one batch, it is a stream of tasks, and the stream version is worth deriving separately because the shape is different. In steady state the lanes deliver work at N/(L+A) tasks per minute and the gate consumes it at 1/A. Throughput is whichever is smaller, so the speedup over one-at-a-time is:

The stream version, and the knee
S_stream(N) = min( N , (L + A) / A ) = min( N , L/A + 1 )
Perfectly linear until the gate saturates, then flat forever. Same ceiling as the batch formula, but a hard corner instead of a gentle curve. The corner sits at N* = L/A + 1, which for a 40-minute lane and an 8-minute audit is 6. Five lanes gets you 5 times. Six gets you 6. Seven gets you 6, and so does seventy.

This is Amdahl's law wearing a different hat, and naming it that way tells you where the leverage is. The sequential fraction of this system is the audit, so the highest-value move is not adding lanes, it is making A smaller, and every bit of that comes from designing claims to be checkable by a command rather than by reading.

Amdahl's law: the speedup of a parallel system is capped by the fraction of work that must stay sequential. Here the sequential fraction is the audit.

That reframing changed how I write briefs. "Report what you changed" produces prose, and prose costs me four minutes to convert into something checkable. "Report the list of changed paths and the suite summary line verbatim" produces something I can compare against reality in thirty seconds. Same lane, same model, same work, and my ceiling roughly doubled because the audit got cheaper. Structured reporting reads like paperwork and behaves like throughput.

Two things the formula leaves out

Rework. Lanes fail their audit, and a failed lane goes back out and comes through the same serial gate a second time. If a fraction f of lanes need that second pass, each delivered task costs 1+f trips through the gate, so throughput at saturation falls from 1/A to 1/(A(1+f)) delivered tasks per minute. Be careful with what that does to the ratio, though: a one-at-a-time baseline reworks at roughly the same rate, so f largely cancels out of the speedup and the ceiling stays near L/A + 1. What does not cancel is absolute throughput, gate time, and the bill. At f = 0.3, three lanes in ten produced nothing shippable and still consumed a full audit each. That is the strongest argument I know for spending twenty minutes on a brief, because f tracks brief quality far more closely than it tracks model quality.

The sentinel's context window. There is a second ceiling that has nothing to do with time. Evidence for one lane costs some number of tokens, call it E, and the sentinel has a finite window W. Once the accumulated evidence approaches W, the sentinel compacts, and a compacted gate is a gate that has started summarising the very details it exists to check. So the real bound is min(L/A + 1, W/E). On long audit-heavy days the second term binds first, which is why the sentinel does not write code: every token it spends on anything else is a lane it can no longer audit properly.

One last effect the arithmetic hides. If all lanes start together they finish together, and then the gate has a queue while every lane sits idle waiting for a verdict. Gate utilisation hits 100% and lane utilisation collapses. Staggering starts, or deliberately mixing lane sizes, smooths the arrivals without changing the makespan of the batch. In practice I pair one long lane with two or three short ones rather than launching four identical ones.

Where your fleet stops paying

The live version of this on the site is a slider: set your own L and A, and watch the speedup bars flatten. The thing to read off them is that past a certain lane count you are spending tokens for almost no wall-clock gain, and the position of that knee is fixed by how expensive your audits are, not by how many agents your machine can run. The model assumes lanes are independent, start together, and never fail, so treat it as optimistic.

Part 03: Rules Written by Failures

06. Rule 2: Worktree Isolation, and the Trap Inside It

Two agents editing the same checkout is a lost-update machine. One writes a file, the other writes the same file from a stale read, and now half of one lane's work is gone with no error anywhere. Worse, they stage each other's half-finished edits, so a commit from lane A contains three broken lines from lane B and the test failure looks like it belongs to A.

The fix is standard and it works: a git worktree per lane, a branch per lane. Each lane gets its own working directory and its own index while sharing one object store, so the filesystem cannot be a shared mutable variable between agents. Two rails go with it. Lanes never run destructive git operations, because the object store, the refs, and the hooks are all shared even though the checkouts are not, so a garbage collection or a force-delete in one lane reaches into every other one. And lane scopes are assigned so that two lanes do not touch the same files, since a worktree prevents corruption but does not prevent a merge conflict you scheduled yourself.

Then there is the trap, which cost me an hour the first time and which I have not seen written down anywhere.

Hidden directories change tool behaviour

My worktrees live under a dot-directory, because that is where the agent harness puts them. Plenty of tools skip hidden directories by convention: indexers, file walkers, some test and coverage discovery. So a couple of tests in my suite, the ones that walk the workspace and assert on what they find, fail inside the worktree and pass on the main checkout. Identical code. Different path visibility.

The first time it happened I spent an hour debugging a defect that did not exist. The rule that came out of it: a lane's green suite is a smoke signal, and the gate's suite on the main checkout is the gate. Every merge decision runs the suite where the code will actually live.

There is a related thing that catches people who set worktrees up correctly and then trust them too far: isolation ends at the merge. Lane 3's suite went green against the main branch as it stood when lane 3 started, and by the time lane 3 reaches the gate, lanes 1 and 2 have already landed. So the suite run that decides the merge has to happen after the merge commit exists, not before it, on the resulting tree. I merge in the order the audits complete, and each merge is followed by its own run. Twice now that post-merge run has caught a pair of changes that were individually correct and jointly wrong, which no amount of pre-merge testing in isolated worktrees would have found.

Dot-directories are only the version of this I happened to hit. Worktrees isolate files and nothing else. Ports, caches, temporary directories, on-disk databases, and background daemons stay shared, and two lanes that each start a service on the same port collide in a way that reads as flakiness rather than as a collision. So the brief either pins a lane's runtime resources to unique values or forbids it from starting them. In my case the expensive shared resource is index building, which is why heavy indexing jobs run strictly one at a time (section 10).

Isolation is per-resource, and you only have the isolation you explicitly built. Everyone remembers files. Almost nobody remembers the port.

07. Rule 3: Tier the Models by Failure Cost

The instinct is to put the best model everywhere. It is the wrong instinct, and the argument against it is not really about money.

Different roles have different failure costs. A coder's mistake gets caught twice: by its own suite and again at the gate. A reviewer's mistake is a false negative, which is invisible and therefore expensive, or a false positive, which is caught when I try to reproduce it. The gate's mistake ships. Those are three different consequences, so they justify three different amounts of capability.

My split: the sentinel runs on the strongest model available, coders on a mid tier, reviewers and validators on a strong but cheaper tier. In current terms that means Opus at the gate, Sonnet in the lanes, and a previous-generation Opus doing review.

Where the cost actually lives

Count agent-runs by role in a typical loop: N coders, two or three reviewers, one sentinel. Coders also read and write the most, so they carry the largest token volume per run. Total spend is roughly:
cost ≈ N · V_coder · p_mid  +  R · V_rev · p_rev  +  1 · V_gate · p_top
The only term that scales with fleet size is the first one. Putting the top-tier model on that term multiplies the fastest-growing part of the bill by roughly five, in exchange for capability on the role that already has two independent checks behind it. Meanwhile the gate, which has no check behind it at all, is a single instance whose price barely moves the total. Buy judgment where there is one of it. Buy throughput where there are many.

The honest cost of tiering down the coders is that mid-tier models need tighter briefs. Ambiguity that a top-tier model resolves sensibly, a mid-tier model resolves plausibly and differently from what you meant. So you pay in specification what you saved in tokens. I think that is a good trade, and not only for the money: a brief is a reusable artifact that makes the next lane better, and tokens are gone the moment they are spent. Being forced to write a precise brief has improved my task decomposition more than any tool has.

What I would not do is tier the reviewers down to the cheapest option available. A weak adversary produces the most dangerous output in the whole system, which is a clean review that means nothing. False confidence at the review stage does not fail loudly; it fails later, in front of a user, and by then you have a track record of green reviews telling you the process works. Reviewers stay strong.

08. Rule 4: Killed Agents Get Resumed, Not Restarted

Agents die. The quota window runs out mid-lane, the API returns an error the harness cannot recover from, the machine sleeps and takes the stream with it. On a long run this is not an edge case, it is a weekly event.

The thing to understand about an agent death is exactly what is lost and what is not. The worktree is on disk. The branch is on disk. Every edit the agent had already written is on disk. A kill loses no disk state. What it loses is context: the plan the agent had formed, the four files it had read and understood, the reason it chose the approach it chose, the thing it discovered in minute twelve that changed the design.

Which means the recovery move is to restore context, not to restart work. Resuming an agent from its transcript brings back the reasoning that produced the half-finished diff. Starting a fresh agent instead produces something worse than a slow restart: an agent with no memory of the edits already on disk, which will re-derive a plan, re-read the files, and quite possibly apply a change that is already there. Double-applied edits are a nasty class of bug because they often still compile.

I have verified this twice on real kills, once when a quota window closed under a running lane and once on an unrecoverable API error. Both agents resumed from their transcripts and finished the task. Resume first, always.

When resume fails, change the framing

If the transcript is gone or the resume does not take, do not tell a fresh agent what the dead one was doing. That produces an agent trying to reconstruct someone else's intent from a description. Point it at the diff instead: here is a branch with partial work on it, read it, and finish it. The diff is ground truth and it is already in the right format for an agent to reason about. "Complete this diff" is a well-posed task. "Continue what the last agent was doing" is not.

This rule also quietly explains why disposable coders work at all. Because a lane's durable output is its branch rather than its conversation, the conversation is allowed to be fragile. Design the system so the fragile part is the part you can afford to lose.

09. Rule 5: Quota Is a Shared Resource, So Everything Is Resumable

Development lanes and paid evaluation runs draw from the same quota window. There is no separate budget for "the experiment" and "the work". A benchmark run that eats the window leaves nothing for the lanes, and four parallel lanes on a busy afternoon can close the window before an evaluation has started.

quota window: a subscription's usage allowance over a rolling period. Once it is spent, everything stops until the window rolls over, whether you were mid-task or not.

Most write-ups of agent fleets do not have this constraint, because they are running on a company card. It changes the design, and mostly for the better, because it forces three habits that turn out to be good practice regardless of who is paying.

Everything is resumable. A run that cannot survive being interrupted is a run you cannot afford to start, because you cannot promise it an uninterrupted window. In practice that means checkpointing to disk after each unit of work, writing results incrementally instead of at the end, and making a re-run of an already-completed unit cheap and harmless rather than something the design forbids.

Work is sequenced, not crammed. Full-factorial experiment matrices are simply infeasible: every additional axis multiplies runs, and runs are the scarce thing. So the design is tiered. Run the headline configuration first, the one whose result would change what you do next. Only if it lands do the secondary arms get a window. Half the matrices I have designed were never worth running once the headline came back.

Explore once, reuse everywhere. The most wasteful pattern in a multi-lane loop is four agents independently discovering the same three facts about the codebase, at full token price, four times. So exploration happens once, in its own lane, and its output is written down as notes that later lanes read instead of rediscovering. This is section 11's mechanism, and quota pressure is what made me take it seriously.

I resented this constraint for about a month and now I would keep it even if someone handed me an unlimited budget. It killed a whole class of plans that would have looked productive for a week and answered nothing.

10. Rule 6: Laptop Physics Is Part of the Architecture

This runs on a MacBook. Not a cluster, not a CI fleet. The machine has opinions and they are enforced by the operating system, not by anything I can prompt my way out of.

Sleep kills long-running agent streams. I did not believe this at first and assumed the stream was dying for some network reason. Then I lined up the kill timestamps against the machine's own wake log and they matched to the second. Every one. A long lane left running while the machine idles into sleep is a dead lane. The fix is caffeinate -is wrapped around long runs, which holds off idle sleep for the duration. It does not cover everything: closing the lid on battery still sleeps the machine, and no flag changes that. Long runs mean lid open and power connected.

caffeinate -is: a macOS command that prevents the machine from sleeping while a given process runs. The -i flag blocks idle sleep, -s keeps it alive on AC power.

What survives the kill is the disk state, which is exactly why rule 4 exists. These two rules are the same rule seen from different ends: the machine will take your agents, so build so that losing one costs you context and not work.

Heavy indexing runs strictly one at a time. Building a semantic index over a large repository is the most resource-hungry thing in my stack. Two of them concurrently do not take twice as long; they thrash memory and the CPU, and everything else on the machine, including the lanes you were trying to accelerate, slows to a crawl. This one I learned by doing it and watching a fan-loud hour produce less than a quiet twenty minutes would have.

Your fleet size is bounded by the machine before it is bounded by the model

Every lane is a process, plus the tool subprocesses it spawns: test runners, compilers, language servers, indexers. Six agents is not six chat sessions, it is six build environments competing for the same cores and the same disk. Long before you hit any interesting limit of the orchestration pattern, you hit thermal throttling and swap. When I plan a fleet I count the heavy subprocesses, not the agents.

None of this is profound. It is the kind of thing that reads as trivia until it eats an afternoon, which is precisely why I put it in the ledger and then in this post. The failure mode that costs the most is the one you attribute to the wrong layer, and "my agent framework is unreliable" is what a sleeping laptop looks like from the inside.

11. Rule 7: Notes Are the Handoff, Transcripts Are Not

When a lane finishes, the sentinel needs to know what it learned. I spent a while reading transcripts for this, which is the intuitive move and cost me more context than any other habit I have had to break.

A finished lane's transcript is long, and most of it is procedural noise: files read and discarded, an approach tried and abandoned, tool output nobody needs again. The signal is a handful of findings scattered through it. Reading all of that into the sentinel's context spends the scarcest resource in the system on mostly-noise, and the sentinel's context is scarce because auditing four lanes is exactly the job that needs room to think. Tokens spent absorbing a transcript are tokens not spent at the gate.

So the protocol is: every subagent writes its key findings as notes to a shared working-memory store before it finishes, tagged with its own identifier, and the sentinel recalls notes instead of reading transcripts. The store is a local daemon that every agent talks to over the same interface, so a note written by a coder in one worktree is immediately recallable by a reviewer in another and by the sentinel at the gate.

daemon: a background process that keeps running and answers requests from other programs. Here it holds the shared note store and answers store and recall calls in milliseconds.

A shared scratch file gets you most of the way and then stops. The difference is retrieval: the sentinel asks a question and gets the four relevant notes, rather than paging in the whole file to find them, and that is the entire saving. Attribution matters for a smaller reason that turns out to bite, which is that a claim needs its source attached before I can weight it, and a flat file loses that as soon as two agents append. Surviving compaction and session boundaries is the part I use most in practice: a finding from Tuesday morning is still there on Thursday when it finally becomes relevant.

A note is still a claim

This is the part I have to keep reminding myself. Moving findings from transcripts into notes makes them cheaper to read, and does nothing whatsoever to make them true. A note saying "the parser handles empty input" is exactly as much hearsay as a report saying it, and it now looks more official because it is short, structured, and attributed. Notes make the gate faster; nothing about them makes it optional. Everything in section 04 applies to a note.

I built the tool this loop runs on, so treat me as an interested party. The mechanism itself is not exotic: any store with relevance-ranked retrieval and per-agent attribution would give you the same shape. The seven-agent test in the team-mode post pushed the idea to its limit, with seven agents building a product coordinating only through shared memory and no other channel. That test is also where the boundary showed up clearly: not one of the defects the reviewers found in that run was present in any of the fifty-two notes. Memory told the reviewers where to look. Running the code did the finding.

Part 04: The Bill

12. Honest Accounting: What This Buys, What It Costs, When Not To

Every methodology post I distrust has the same shape: here is my process, it is great, adopt it. So here is the other side.

What it buys

Parallelism that survives the merge. Four lanes converging on a clean main branch, rather than four branches and a weekend of reconciliation. That is the headline and it is real, within the ceiling from section 05.

Adversarial coverage a single agent will not generate. A model asked to check its own work is trying to complete the task "review this favourably-written summary of my work". A separate agent with a hostile brief and no ownership of the diff finds different things. Splitting review across independent axes finds more still, because a single reviewer with three mandates stops at the first easy failure.

Auditability. Months later I can answer why a thing was merged, on what evidence, and what the reviewers said. Not from memory and not from a chat log I would have to re-read, but from a file that was written at the time.

What it costs

The gate is the bottleneck, by design. This is not a flaw to be optimised away; it is where the trust in the system is manufactured. But it does mean the sentinel is the constraint, and on a busy day I am reading evidence rather than thinking about the product. Section 05 quantifies exactly how much that costs you and where it stops being worth it.

Brief-writing is real work. A good lane brief takes ten to twenty minutes, which on a short task is longer than doing the task. That overhead is the reason the pattern has a floor below which it makes things slower, not faster.

Token spend scales with the fleet. Four lanes is roughly four times the tokens, and some of those lanes will produce work you throw away. Tiering (section 07) blunts this. It does not remove it.

Coordination failures are subtle. Two lanes with overlapping scope, a shared runtime resource nobody isolated, a brief that was ambiguous in exactly one place. These do not announce themselves. They show up at the gate as work that has to be redone, and redone work is the most expensive failure mode in the system: it consumed a full lane, produced nothing, and it comes back through the same serial gate, which is the f term from section 05 spending your scarcest minutes twice.

When not to use it

Single-lane work. One focused change, one agent, direct supervision. The whole apparatus is overhead when there is nothing to coordinate. I still use a plain single-agent session for most of my day.

Exploratory work where the spec is unknown. You cannot brief a lane on a task you cannot specify, and a lane briefed vaguely will produce something confidently wrong at speed. Exploration wants one agent and a human who can change their mind mid-sentence.

Anything where the audit costs as much as the task. If A approaches L, the ceiling from section 05 drops to roughly two times regardless of fleet size, and you are running a fleet for a doubling. Do it yourself.

The honest version of the thesis

This pattern does not make agents trustworthy. Nothing in it improves a single agent's reliability by one percentage point. What it does is make their unreliability cheap: bounded to a lane, caught at a gate, recorded in a ledger, recoverable by resume. That is a smaller claim than "multi-agent development works", and it is the one I can actually defend.

Summary

The count of nine that turned out to be seven is a small story, and I keep telling it because everything else in this loop is downstream of it. Not "the model was wrong" but "I merged on a summary". Once you accept that a report is the weakest evidence in the system, the roles fall out, the gate falls out, the ledger falls out, and so does the uncomfortable conclusion that the bottleneck is supposed to be you.

The sentinel writes no product code. It scopes lanes, audits evidence, owns the append-only ledger, and is the only place paid or irreversible actions can originate.
Coders are disposable, one task each, isolated in a git worktree branch, suite green before reporting, in a report shape the brief specifies. Reviewers attack on independent axes, in capped rounds, and their findings are verified before they gate anything.
Rule 1, enumerate. A lane reported nine; the artifacts said seven; the nine had already reached the ledger. Reports are hearsay, artifacts are evidence, and verification has to cross an agent boundary to count.
The gate sets the ceiling. With lanes of length L and audits of length A, batch speedup is N(L+A)/(L+NA) and stream speedup is min(N, L/A + 1). Both cap at L/A + 1. Rework at rate f costs you 1+f trips through the gate per delivered task, and the sentinel's context window imposes a second, independent bound. Cheaper audits raise the ceiling; more lanes do not.
Rules 2 to 4, mechanics. Worktree isolation is per-resource and dot-directory worktrees change how path-walking tools behave, so the gate re-runs on the main checkout. Tier models by failure cost, since cost scales with the fleet and the fleet is coders. Killed agents get resumed from their transcripts, because the kill takes context and never disk state.
Rules 5 to 7, constraints. One quota window funds dev and evaluation, so everything is resumable and exploration happens once. Sleep kills streams (verified against the wake log) and heavy indexing runs one at a time. Subagents hand off through attributed notes, not transcripts, so the gate's context stays free for auditing.
The honest bill. You buy parallelism, adversarial coverage, and auditability. You pay in gate overhead, brief-writing, and token spend, and you accept that the orchestrator is the bottleneck by construction. Below a certain task size this makes you slower.

If you take one thing from this: design your loop so that the cheapest step is the one that catches errors. Everything else is scheduling.

The Agent Never Chooses to Remember: Memory as a Harness Property

Swapnanil Saha — Tue, 04 Aug 2026 19:58:52 +0000

The Agent Never Chooses to Remember: Memory as a Harness Property

Every shipped memory design asks the model to decide to save and to recall. In a control run with the answers already sitting in the store, it decided zero times across 114 turns. Here is the tier we skipped, and how to build it.

I spent most of a year building a memory system for coding agents and only realised at the end that I had been solving the wrong half of the problem. The store was fine. The retrieval was fine. What I had never questioned was the step in the middle: that the agent would decide, on its own, to use either of them.

Open any agent memory product and you will find the same shape. A place to put things, a way to get them back, and a paragraph of instructions telling the model when to do both. Instruction files, memory directories, memory tools over MCP, vector stores with a recall endpoint. The architectures differ wildly. The assumption underneath them does not: somewhere in the loop, the model has to want to remember.

That assumption is the thing I want to take apart. Not because models are lazy, and not because the tools are bad, but because we borrowed the wrong half of human memory when we designed them. The memory that carries you through a working day is not the memory you go looking for. It is the memory that arrives because the situation summoned it. That tier, in an agent, cannot be a skill the model exercises. It has to be a property of the harness, the program the model runs inside, which owns the context window and can put things in it without asking.

This post is that argument, the runs behind it, and the shape of the thing you actually build. A warning up front on scope: the measurements are one task inside one harness at one version, and section 12 is where I lay out how far I think they generalise and what would make me abandon the position.

The paper this post accompanies. The full treatment, including the protocol, the graders, and the two-tier theory this post argues informally, is Delivery, Not Storage: Cue-Anchored Working Memory as a Harness Property for Coding Agents (arXiv:2607.20972, 23 July 2026). Every number in this post comes from that paper. Where the paper hedges, this post hedges the same way.

A note on vocabulary before we start. A harness is the program the model runs inside. It owns the conversation, decides what text enters the context window, runs the tools, and handles compaction. Claude Code, Cursor, and any agent runtime are harnesses. The context window is the fixed-size block of text the model can see on any one forward pass; everything the model "knows" about the current session is either in the weights or in that window.

Part 1: The Assumption Nobody States

01. One Assumption, Every Agent Memory Product

Line up the current designs and sort them, not by how they store things, but by who decides a stored fact enters the model's context window. That column has only three values in it, and the difference between them turns out to matter more than everything the marketing pages compare.

Design	Who decides it arrives	When it arrives
Instruction file	The harness	Session start, and after every compaction. Unconditional.
Memory directory	The model	Whenever the model decides to open it. Possibly never.
Memory tool over MCP	The model	Whenever the model decides to call it. Possibly never.
Vector store with recall	The model	Whenever the model forms a query. Possibly never.
Path-scoped rule file	The harness, conditionally	When a matching file is read. Cue-triggered.

Notice which rows have the sophisticated engineering in them and which rows actually deliver. The instruction file is a flat text document with no retrieval logic whatsoever, and it is the single most reliable memory channel any coding agent has. Not because it is well designed. Because a mechanism outside the model pastes it into context whether the model was thinking about it or not.

The rows below it are where all the interesting engineering happens: retrieval by meaning rather than keyword, reranking of candidates, knowledge graphs, merging old notes into newer ones, ageing out what is stale. Every bit of that quality gets multiplied by a number nobody puts on the slide, which is the fraction of the time the model actually invokes the thing. If that number is near zero, everything above it is arithmetic on zero.

The uncomfortable ranking. Sort agent memory channels by reliability and you get an inverse ranking of engineering effort. The dumbest channel, a static file the harness pastes in, is the most reliable. The smartest channel, semantic retrieval with a well-tuned index, is the least reliable, because it is gated on an act of model initiative that may not happen. Reliability here is not a property of the retrieval. It is a property of who pulls the trigger.

This is not an argument that instruction files are good. They are terrible in most ways: static, unconditional, growing without bound, and paid for on every single cycle. It is an argument that delivery is doing the work we keep crediting to storage and retrieval. And once you see that, the next question is obvious: is there a way to get instruction-file reliability without instruction-file cost? That question is the rest of this post.

02. The Memory Tier You Do Not Choose

Try to remember what you had for lunch on the third Tuesday of last month. You just did something effortful: you constructed a search, you probed for constraints, you either found it or gave up. Psychologists call that voluntary retrieval. It is slow, it is strategic, and it is the tier every agent memory tool models.

Now think about the last time a smell in a stairwell put you somewhere twenty years ago, unbidden, complete, before you had decided anything. Or, less romantically: you sit down to edit a config file and the thought careful, this one is load-bearing, someone broke prod with it in March shows up on its own. You did not query for that. The situation delivered it.

That second tier is involuntary retrieval, and diary studies of everyday cognition find it is not a curiosity at the edges. It is a routine, frequent part of how people function, and it shows up most in situations where attention is diffuse, which is to say, while you are busy doing something else. The retrieval is associative and context-sensitive rather than effortful and strategic.

The condition for it firing is well characterised. Endel Tulving's encoding specificity principle says a cue works to the degree it overlaps with what was encoded when the memory formed. Later work on spontaneous retrieval sharpens it: distinctive cue-to-event pairings produce far more involuntary memories than generic ones. A specific sound tied to a specific scene brings the scene back. A generic sound brings back nothing in particular.

The two tiers, in a kitchen.

Voluntary retrieval is looking up a recipe. You know you need it, you know roughly where it lives, you go and get it. It works, and it costs you a deliberate trip.

Involuntary retrieval is reaching for the pan handle and your hand stopping short, because that handle was hot once. Nobody looked anything up. The situation, the handle, the reach, produced the memory as a side effect of being in that situation. If you had to consciously query "is this handle historically hot" before every reach, you would eventually forget to, and you would eventually get burned.

Here is the part that matters for design. The tier that keeps you safe while you work is the second one, and it is precisely the tier that is not under your control. You do not decide to remember the hot handle. Deciding is the failure mode. If the memory required a decision, the decision would compete with the actual task, and the actual task would win, because the actual task is what you are attending to.

Every agent memory product I have used models the recipe book. None of them model the hot handle. And the hot handle is where the load is.

What this analogy does not claim.

I am not saying a language model has episodic encoding, consolidation, or anything resembling involuntary recall as a cognitive process. It does not. The model has a context window and a set of weights, and that is the whole of it.

What transfers is the delivery discipline, not the mechanism. Human memory research earns its place here for one reason: it establishes that a system doing serious work leans heavily on a retrieval path the agent never initiates, and that the path is governed by cue specificity. That constraint you can implement in a harness with a glob pattern and an event hook. Take the constraint; leave the neuroscience.

One caveat before anyone who knows this literature objects. Human involuntary recall is not reliable. It fires at useless moments, it misses the moments you needed it, and it is famously bad at surfacing the name of the person walking towards you. If the argument were "copy human memory", that unreliability would be a serious problem for it.

It is not the argument. A cue in a harness is a boolean over observable state, so it fires exactly when it is true and never otherwise. The engineered version is strictly more dependable than the thing that inspired it, and that is the point of building it rather than admiring it. What biology contributes is the observation that the un-initiated path is load-bearing at all, which is not obvious and which the entire agent memory field has been building as though it were false.

Part 2: What the Runs Actually Showed

03. Zero Voluntary Memory Operations in 114 Turns

Theory is cheap. The reason I believe the two-tier framing is a run I set up to disprove the opposite hypothesis, and could not.

The setup removes every excuse an agent could have for not using a memory store. The store was pre-seeded with four notes about the exact task the agent was about to attempt: real gotchas, the kind of thing that saves a wrong turn if you read it. The memory tools were connected and verified in the session's own initialisation. The workspace instruction file carried usage guidance telling the model when to reach for them. Then the agent was given a naturalistic coding task and left alone.

The measurement. Across 114 turns, the agent performed zero voluntary memory operations. It never wrote a note. It never read one. The four answers to its own problems sat in the store the entire time, one call away, unqueried.

A turn here is one full think-act-observe cycle: the model produces a response, optionally calls tools, receives the results. One hundred and fourteen of them went by. The agent read files, ran commands, wrote code, and made progress on the task. It simply never treated the memory store as part of its working repertoire.

The first thing anyone says when I show them this is that the guidance must not have been strong enough. It is a fair reflex and the numbers do not support it. Across the five unseeded tool-equipped runs in the same set, voluntary memory writes came in at zero to one per run, against 32 separate mentions of memory guidance in the workspace instruction file that the model reads at every session start. A different run in the same family put ten tools in front of the agent with 30 mentions of them in that file and got zero tool calls in 63 turns. There is no dose-response curve here to climb. More words did not buy more calls.

The second reflex is that this is a bias against third-party tooling, which is a more interesting hypothesis and also wrong. The harness ships its own native memory feature: a built-in directory the model is trained to write notes to itself in, no external server involved, no tool schema to learn. That directory was never created in any run of any arm. So the model was not preferring the house option to mine; it was declining the whole category of stopping mid-task to do memory hygiene.

The third reflex, that the task simply was not hard enough to need memory, is the one that took a whole extra experiment to rule out, and section 05 is where that goes. It is the strangest result in the project.

The confound I have to state before you quote this.

This is one configuration, not a law. The tool descriptions, the guidance wording, the system prompt, and the task shape were all held fixed at one setting, and incentives were not manipulated. A different tool description, a stronger prompt, or an explicit reward for using memory might well produce nonzero usage; I have not run those arms.

So the claim is deliberately narrow: this configuration produced zero. What that licenses is not "agents cannot use memory tools". It is "you cannot assume they will, and if your design's reliability depends on them doing so, you have an unmeasured dependency sitting at the centre of it".

That narrower claim is still enough to break most designs, because most designs do not have a fallback. If the recall does not happen, nothing happens. There is no partial credit for a store with the right contents.

04. What the Compaction Probe Adds to the Picture

The zero tells you the voluntary channel does not fire. It does not tell you whether that matters, because maybe the conversation holds everything the agent needs anyway. The second experiment closes that hole.

Compaction is what happens when a long session fills its context window. The harness takes the older conversation, has the model summarise it, and replaces the raw history with that summary. It comes in two flavours: autocompact, which fires on its own at a threshold you do not control, and the manual /compact, which fires when you ask and lets you pass a custom instruction into the summarisation prompt.

People reach for that custom instruction as the fix, and it is worth understanding why it does not hold. Writing "preserve all operational facts" into the compact prompt is still asking a model to decide what counts as an operational fact, on a context it is seeing under compression pressure, with no way for you to check the result before the original is gone. You have not removed the judgement call. You have moved it one step and made it harder to audit.

The structural catch is worse than the judgement call. Each compaction summarises the previous summary, not the original conversation. Summary k is a function of summary k minus one plus whatever happened since. So a detail that one pass declines to carry cannot reappear in any later pass, because there is no original left for it to come from. The only way a lost fact returns is if something outside the recursion puts it back into the raw context, which is exactly what happened in the two anomalous summaries below, and exactly what a delivery hook does on purpose.

The probe seeds ten synthetic operational facts into a session exactly once, deletes the file they came from so they cannot be re-read, then forces compaction over and over and grades every summary for whether the facts are still there. Two arms, identical workload, differing in one thing: one has no memory tier, the other has the same ten facts delivered back into each fresh context by harness hooks.

The two channels, measured.

Conversation-only facts: absent from 106 of 108 post-compaction summaries. The two exceptions are the re-entry case: the agent happened to grep something that put the facts back into raw context, they survived roughly one further summarisation pass, and then they were gone again. No summary ever carried a partial set. Always ten or zero.

Harness-delivered facts: arrived at all 138 compact-resumes of the memory-equipped arm. Delivery fired at every fresh window and nowhere else.

Those two numbers measure different channels, and conflating them is the most common mistake I see people make with this result. The 106-of-108 figure is about the summary: what the summariser chose to carry forward. The 138-of-138 figure is about delivery: what the harness put into the fresh window afterwards. In the memory-equipped arm the two came apart completely: from summary 67 onward there is a stretch of 72 consecutive summaries carrying none of the ten facts, while deliveries kept arriving on schedule throughout, and the run was entirely unaffected because the facts were in the window regardless of what the summary said about them.

Being in the context is not being in the summary. In the memory arm, the very first summary dropped all ten facts even though they had been injected into the exact context the summariser was compressing. The summariser is a model making a fresh judgement about what matters, every time. Whether it keeps your fact is not something you can design around, and in this run it flipped mid-way for reasons I still cannot fully explain. A delivery channel does not have that problem because it never asks.

Put the two experiments together and you have the shape of the argument. Facts that live only in the conversation die at the boundary. The voluntary channel that could bring them back does not fire. Therefore, in any session long enough to compact more than once, a fact that is neither in an instruction file nor delivered by a hook is a fact that is functionally gone, no matter how well you stored it.

05. The Agent That Grepped Its Own Session Logs

My favourite result in the whole project was not designed. It fell out of the no-memory arm and it is the reason I stopped worrying that the demand for this tier might be imaginary.

In the closing phase of that arm, the facts had been gone from every summary for dozens of boundaries. The prompt asked for them in the final report. The agent's shell access was explicitly forbidden, and that ban was restated in every single phase prompt. It used the shell anyway. It worked out that the harness persists its own session transcript to disk as newline-delimited JSON, and that the CLI keeps tool results as files too, and it grepped those artifacts for the fact markers until it found them. Of the 260 shell commands it ran, 38 were this archaeology.

Read that as a capability statement and it is genuinely impressive. Denied a memory tier, the model reverse-engineered one out of the substrate it happened to be standing on. It understood that the harness's disk state outlives its conversation state, and it mined the difference.

Read it as evidence and it says something narrower and more useful: the need was real. This was not a system politely declining a feature it had no use for. It was a system that needed the information badly enough to violate a restated constraint and burn 38 shell commands reconstructing it. The demand exists. What is missing is a supply path that does not depend on the model noticing it needs one, mid-task, under load.

Need and initiative are different variables. The same run gives you both halves. Given a memory store and a tool, the model does not call it. Denied a store entirely and pushed to produce the facts, the model builds its own retrieval out of log files. Those are not contradictory. Voluntary recall is a deliberate act competing with the task; log archaeology at the end was the task. The gap between "needs the information" and "will go and get the information at the moment it would help" is exactly the gap a delivery tier fills.

There is a smaller lesson here that cost me a pilot run, and I include it because it is the kind of thing that will silently invalidate your version of this experiment too. The harness restores recently-read files across the compaction boundary, small ones verbatim. In an early version of the probe, the fixture holding the ten facts came back whole right after the first compaction, the agent recited all ten, and for about an hour I believed the facts had survived. They had not; the file had. That is why the real runs delete the fixture before forcing anything, and it generalises: when you measure memory, make very sure you are not measuring file restoration wearing memory's clothes.

Part 3: Building the Tier

06. Stored Is Not Available: The Design Axiom

Here is the sentence the whole project reduces to, and I would put it on the wall of any team building this:

The axiom. Availability is a property of the delivery rule, not of the store.

Storage answers "does this fact exist somewhere". Availability answers "is this fact inside the model's context at the moment it changes what the model does". They are independent. A store at 100 percent storage and 0 percent availability contributes exactly nothing, and it will look healthy on every dashboard you build for it, because dashboards measure the first quantity.

Most memory evaluations I have read measure retrieval precision on held-out queries. That is a measurement of the store, conditional on a query arriving. It says nothing about whether the query arrives. If you want a number that predicts value, you want availability, and you can write it down.

The metric to instrument.

For a fact f, let R(f) be the set of turns at which f is relevant, meaning it would change what the agent does. Let C_t be the context window at turn t. Then
A(f) = |{ t in R(f) : f in C_t }| / |R(f)|
Availability is the fraction of the moments that mattered at which the fact was actually present. For a voluntary tier, A(f) factors as P(the model issues a query) times P(the store returns f | queried). Store-side work only improves the second factor. The measured value of the first was zero, and zero times anything is what it is.

For a delivered tier the first factor is replaced by whether your cue predicate is true at those turns, which is something you control at design time and can check by inspection.

I would push this metric on anyone benchmarking agent memory. Retrieval precision on a query set is a number about your index. Availability is a number about your product. They can differ by a factor of infinity, and in the seeded control they did.

Interactive demo 1: stored versus available

(Live on the canonical post.)

One note, one session, three tracks. The top track shows what the store holds, which never changes. The two below show whether that note is actually inside the model's context at each turn, once under a voluntary tool the model may or may not call, once under cue-anchored delivery. Drag the call rate down to the measured zero and watch the second track go dark while the store stays full.

The rules the simulation uses: a compaction clears the window, so anything that was in context is lost unless it is delivered again; a voluntary call at rate p puts the note in context from that turn until the next compaction; cue-anchored delivery fires at session start, after each compaction, and when a file matching the note's path glob is touched. Mechanism illustration, not measured data.

The thing to take from it is not that one track is greener. It is that the top track never moves. Everything inside the store, how the text is split up, how it is indexed, how results are ranked, how old notes are merged or aged out, is a decision about track one. Tracks two and three are decided by something else entirely, and that something else is where reliability lives.

07. What a Cue Actually Is, Mechanically

"Cue-anchored" sounds like a metaphor. In an implementation it is not; it is a boolean.

Definition. A cue is a predicate over harness state, evaluated by the harness at fixed lifecycle points, whose truth causes a stored note to be written into the model's context.

Three properties are doing all the work in that sentence. It is evaluated by the harness, so no model initiative is required. It fires at fixed points, so it is deterministic and auditable. And it is a predicate, so it can be as narrow as you like, which is what stops the channel from degenerating into a second instruction file.

The state a harness can see is richer than people assume. It knows the session just started. It knows a compaction just finished. It knows which file is about to be edited, which command is about to run, what the user typed in the prompt that is about to be submitted, and what time it is. Each of those is a candidate condition.

The condition classes worth having

These are the ones I ended up shipping in vectr, and the ones I would build again from scratch:

Path glob. The note fires when a file matching the pattern is touched. This is the workhorse. Most operational knowledge is anchored to a file.
Lifecycle event. session-start, prompt-submit, pre-edit, pre-run, pre-commit, post-compaction. These are moments the harness already owns, which is why riding them costs nothing structural.
Symbol reference. An exact symbol name, so a caveat about one function surfaces when that function is in play rather than whenever its file is opened.
Semantic match. Similarity between the note and the submitted prompt. This is the one condition class that is probabilistic rather than exact, and it is the one to be most careful with.
Temporal guard. not_before, expires_visibility, cooldown. Not for finding the note but for governing how often it is allowed to arrive.

The event surface is coarser than that list makes it sound

Worth saying plainly, because it is the first thing you hit when you implement this. Those six event names are my vocabulary, not the harness's. Claude Code exposes SessionStart, UserPromptSubmit, PreToolUse, PostToolUse, PreCompact and a few more. My pre-edit and pre-run are both PreToolUse with a matcher on the tool name. My pre-commit is PreToolUse on the shell tool with a pattern match for a git commit, which is leaky in the obvious way: a commit made through some other path does not look like a commit to the matcher, and you will not find out until the note fails to fire.

Where the output lands also differs by hook, and it reads differently to the model. UserPromptSubmit writes to standard output and the harness prepends that text to the user's prompt, so it arrives looking like part of what the user said. PreToolUse feedback arrives attached to the tool call. Same bytes, different framing, and in my experience the prompt-prepended channel gets attended to noticeably more. That is an impression from watching transcripts, not a measurement, and I would like someone to run it properly.

Composition, and why the semantics matter

Conditions combine with AND inside a single trigger entry and OR across entries. That is a small design decision with a large consequence: it lets you narrow a fire by conjunction and cover multiple distinct moments by disjunction, without a query language.

vectr_remember(
  content="tests/test_session.py deadlocks under parallel test execution. "
          "Run it with -p no:xdist or it hangs the whole suite.",
  kind="gotcha",
  triggers=[
    {"event": "pre-edit", "path": "tests/test_session.py"},   # entry 1: AND
    {"event": "pre-run",  "path": "tests/**"},                # entry 2: AND
    {"event": "pre-commit"},                                  # entry 3
  ],
)
# fires when: (editing that exact file) OR (running anything under tests/) OR (committing)

Most notes never need an explicit triggers list, because the note's kind already implies a delivery rule. That mapping is the part of the design I would defend hardest, because it is what makes the tier usable by an agent that is not thinking about delivery at all.

Kind	What it holds	Default delivery rule
`directive`	A standing rule the user stated once	Injected at every session start, unconditionally. This is the one class that behaves like an instruction file, on purpose.
`gotcha`	A caveat anchored to a specific file	Fires when that file is touched. Silent otherwise.
`task`	Current-work state	Returned newest-first, because for in-flight work recency beats similarity.
`finding`	A learning about the codebase	Relevance-ranked against what is being worked on.
`reference`	A pointer to a URL or ticket	Retrieved on demand. Low value delivered, high value looked up.
`decision`	An architectural decision and its reasoning	Not auto-injected. Recalled chronologically when someone asks for the decision history.

Look at the last two rows. Not everything belongs in the delivered tier, and a design that pushes everything into it has misunderstood the point. The question for each note is not "is this valuable" but "can I name the moments at which it changes behaviour". If you can name them, deliver it. If you cannot, it is a lookup, and a lookup is fine.

08. Composing Triggers: Precision Is the Whole Game

Cue design has the same structure as any retrieval problem, with one twist that changes the economics. A cue that never fires misses the moment. A cue that fires everywhere costs tokens on every cycle, and unlike a bad search result, you cannot ignore it: it is already in the context, already paid for, already displacing something else.

Interactive demo 2: design a trigger

(Live on the canonical post.)

One note ("tests/test_session.py deadlocks under parallel test execution, run it with -p no:xdist", about 42 tokens), a fixed stream of eighteen harness events, and six of those events are moments where the note would actually change what the agent does. You compose up to three trigger entries out of an event selector, a tests/** path condition, a semantic condition on the prompt, and an optional cooldown. The demo scores moments caught, moments missed, surplus fires, and injected tokens.

The naive preset, one entry matching any event, costs 756 injected tokens on a stream of eighteen events, for a note worth 42. That is what "just inject it always" looks like once you count, and it is roughly what happens to every instruction file that has been alive for a year.

The tuned preset catches all six moments and still leaves one surplus fire, and that one is the more useful lesson. Running pytest tests/test_billing.py and running pytest tests/test_session.py are different situations to you and identical situations to a glob over tests/**. You cannot cue more finely than your harness can observe. Every cue design bottoms out on that ceiling, and knowing where yours sits is most of the work.

The asymmetry between a miss and a surplus fire.

They are not symmetric errors, and the direction flips depending on scale. One miss costs you one bad decision, once. One surplus fire costs you tokens on every cycle the cue matches, forever, and that cost compounds across a session while the miss does not.

Which means the usual retrieval instinct, favour recall and let ranking sort it out, is wrong here. In a delivered tier, over-firing is the failure mode that kills the configuration, and section 09 is about why that is more literal than it sounds.

09. What Delivery Costs, Honestly

I have spent eight sections arguing for delivery. Here is the bill, because a post that only showed the wins would be lying by omission.

Fixed per-cycle context, not a one-time payment

Every delivered token is a token of context you do not get back. In the probe, each delivery cost around 240 tokens, mean injected text of 969 characters under the standard four-characters-per-token estimate, and roughly 34k tokens across the full run. As an absolute number that is small. As a structural property it is not, because it is paid on every cycle rather than once.

That distinction bit me in a way I did not anticipate, and it cost me two invalidated launches before I understood what I was looking at. Claude Code has an autocompact thrash guard: it aborts the call when the context refills to the limit within three turns of a compaction, three times running. Sensible protection against a session that has become a compaction loop. The guard is not memory-aware and should not have to be, because all it reads is refill pressure. But a memory-carrying configuration has fixed per-cycle payload the bare one does not: reloaded tool schemas, guidance files, per-prompt injections. None of it large. All of it always there. At the shared file-read cap both arms started with, the guard killed the memory arm's first audit phase fourteen turns in. Running that arm at all meant dropping its read cap from 8k to 6k, and even then the guard killed the final phase once the growing report had eaten the remaining headroom.

Lean payload is not a nicety, it is a precondition. A harness tuned for memoryless operation treats a memory-carrying configuration's context profile as elevated thrashing risk. Your per-cycle payload competes for exactly the headroom the guard is watching. Roughly 240 tokens per delivery was what let the configuration run here. That number will differ in your harness; the fact that there is such a number will not.

Costs that only show up once you run this for a while

Duplication is the one I did not see coming. A delivered note enters the context; the next compaction summarises that context; if the summariser happens to keep the note, it is now in the summary as well. Delivery then fires again on the fresh window and you are carrying two copies, neither of which knows about the other. Either you deduplicate against what is already present, or you pay for the note twice for the rest of the session. Nobody I have read on agent memory mentions this, and it falls out of the design immediately.

Then there is latency, which is a design constraint disguised as an implementation detail. Hooks run synchronously in front of the operation they are attached to, so a cue evaluator on pre-edit means every edit the agent makes now waits on your predicate. Make that a network round trip to a hosted store and you have slowed the agent down on every file it touches, all day, in exchange for a note it may not even need. That is a large part of why vectr runs as a local daemon with recall under 50ms instead of as a service. Not a purity argument about local-first software. The budget for a synchronous pre-edit hook is simply tiny, and the architecture has to fit inside it.

The one I would flag hardest to anyone building this is that the directive class grows without bound. It is the kind that injects unconditionally at every session start, which means it behaves like an instruction file and inherits the instruction file's disease exactly. Every standing rule anyone adds stays forever. The block only ever gets bigger. Eighteen months in you are back where you started with more moving parts. So the tier needs a budget: a hard token ceiling on the unconditional block, with the least-recently-relevant entries evicted when it is exceeded. I would much rather be forced to demote a stale directive into a cued one than discover the block has quietly reached four thousand tokens.

The rest of the ledger

The memory arm ran 21 percent more turns and 36 percent more cost than the baseline. Some of that is delivery overhead, some is the tighter read cap the guard forced on it, and some is a late-run stretch where the agent began inventing progress it had not made, announcing phases and file ranges that did not exist. Its audit completeness, scored against a mechanical count over the files it was supposed to have reviewed, came in at 0.70 against the baseline's 0.93. I would not read that gap as a memory effect. It is confounded with the read cap, and the read cap is the thing I had to change to get the arm to run at all.

Which is worth sitting with for a second, because it is not the result I wanted. The memory arm was slower, more expensive, and scored worse on the task metric. What it did do was keep ten facts alive across 138 compaction boundaries that the baseline lost at the second one. Those are answers to different questions. If your session is short enough never to compact, this tier is pure overhead and you should not build it.

The failure mode that worries me most

Not cost. Staleness. A delivered fact that has gone wrong is worse than an absent one, because the agent has no way to tell. An absent fact produces uncertainty, which is a state the model handles reasonably. A confidently delivered wrong fact produces confident wrong action, and it arrives with the authority of the harness behind it.

So revocation has to be a first-class operation, and deletion is not good enough. If you silently delete a note the agent has been acting on, the belief it seeded does not disappear with it. What worked better was keeping a revoked note visible as a correction: it still surfaces on recall, marked as previously believed and now wrong, with the reason attached. The agent sees the retraction rather than a hole where a fact used to be. Temporal guards do the softer version of the same job: expires_visibility on a note about an in-flight migration stops it outliving the migration.

Three ways this tier goes bad in production.

Over-firing. Broad globs and generous semantic thresholds turn the tier into a second instruction file, at which point you have reinvented the thing you were trying to improve on, with more moving parts.

Staleness. Delivered facts age. Without expiry and revocation, the tier degrades from useful to actively misleading, and it does so invisibly.

Semantic cues that drift. The one probabilistic condition class is the one that silently widens as the store grows. Every other class is exact and stays exact. Threshold it conservatively and prefer the exact classes where you can.

10. Where Voluntary Retrieval Still Belongs

I want to be careful not to argue past what the evidence supports, because the obvious misreading of this post is "delete the memory tools". That is not the conclusion, and if you act on it you will build something worse.

The voluntary tier is the right shape for anything whose cue you cannot enumerate in advance. Exploring an unfamiliar area of a codebase. Answering a question nobody anticipated. Pulling up a decision history because someone in review asked why the schema looks like that. In every one of those cases the agent has a query and no way for the harness to have predicted it. Retrieval is exactly right there, and a good index earns its keep.

What the measurements rule out is one specific and very common bet: putting load-bearing facts behind a call the model has to remember to make. The model is perfectly capable of making that call. The problem is that making it competes for attention with the task, and the moments you most need the fact are precisely the moments the task is most demanding. You are asking for initiative at the exact point initiative is scarcest.

Question	Delivered tier	Voluntary tier
Cue known in advance?	Yes, that is the precondition	No, the query is formed at use time
Failure if it does not fire	Silent wrong action	The agent explores instead. Slower, not wrong.
Cost model	Fixed, per cycle, paid whether used or not	Paid only when invoked
Right content	Standing rules, file gotchas, live task state	Open-ended lookup, history, references
Right size	Small. Tens of notes, tight cues.	Large. The store can grow indefinitely.

The two tiers are complements with different economics, and the mistake the field keeps making is not building the wrong one. It is building only one, calling it memory, and being surprised when the durable facts do not survive.

Part 4: Provenance and Limits

11. This Idea Has Ancestry, and I Should Say So

Cue-triggered retrieval is not something I invented, and a post that implied otherwise would be worth less than one that traces where it came from.

Production-rule systems have worked this way since the 1970s. A production rule is a condition-action pair, and the system fires it when its condition matches working memory. Nothing calls the rule. The match is the invocation. That is structurally the same move I am describing, applied to knowledge delivery rather than action selection.

ACT-R goes further and is worth reading if you build this. Its declarative memory retrieves chunks by activation, and activation spreads from whatever is currently sitting in the system's buffers. The context you are in changes what is retrievable, without any explicit retrieval request. Soar makes a related move, biasing retrieval from declarative memory using the activation of elements already in working memory. Both architectures took cue-driven retrieval as foundational decades before anyone was writing hooks for a coding agent.

And the coding harnesses already ship pieces of this. Claude Code re-reads and re-injects the project instruction file after every compaction, unconditionally, with no model involvement. Path-scoped rule files reload when a matching file is read, which is a path-glob cue in everything but name. The mechanism exists. What it lacks is a way for a note the agent wrote during the session to opt into the same delivery.

What is actually new here. Three things, none of them the cue idea itself. The composition: one store where a note's kind implies its delivery rule, and explicit triggers compose path, event, symbol, semantic, and temporal conditions over that default. The delivery argument: the claim that for the operational tier this is not an optimisation but the only channel that works. And the measurement: an actual number for what the voluntary channel delivers in a controlled run, which as far as I can tell nobody had published.

The measurement is the part I would defend most. Everyone in this field has an intuition about whether agents use their memory tools. Intuitions are free. The seeded control cost real money and produced a number, and the number was zero.

12. What Would Change My Mind

The evidence here is narrower than the thesis, and pretending otherwise would make the thesis less useful, not more.

One task, one harness, one model. The compaction probes and the adoption runs are a single naturalistic coding task inside Claude Code CLI v2.1.211 in July 2026, driving Claude Haiku 4.5. Harness internals are exactly the kind of thing that changes between point releases, and I would not be surprised to find the thrash guard retuned or the file-restoration behaviour different by the time you read this. Read the specific mechanisms as a dated snapshot. The shape of the finding is the part I expect to hold: a channel gated on model initiative has a reliability you cannot design, and a channel gated on a harness predicate has one you can.

A smaller version of the same caveat: a more capable driving model might well volunteer more. Haiku 4.5 is a fast, cheap model and it is a fair question whether a frontier model with more headroom would spend some of it on memory hygiene. I would guess it helps at the margin and does not change the structure, since the competition for attention gets worse, not better, as tasks get harder. That is a guess and I have labelled it as one.

The zero has an incentive confound. Tool descriptions, guidance wording, and task framing were fixed at one setting, and the agent was never given an explicit incentive to use memory. Elicitation arms that vary those, a differently phrased tool description, a system-prompt nudge, an explicit reward, are the obvious next experiment and I have not run them. If one of those arms produced substantial voluntary usage, the correct update would be: the voluntary channel is elicitable, and the design question becomes how robustly, across which models, at what cost to the task.

Notice that even a strong elicitation result would not restore the original assumption. A channel that works when you tune the prompt just right, on this model version, is still a channel whose reliability you have to re-establish on every model update. The delivery channel does not have that property, and that asymmetry is most of why I favour it.

The falsifier I would take seriously. A run where an agent, without harness-side injection, reaches a comparable availability number on operational facts across many compaction boundaries, using only voluntary recall, and does it across model versions rather than on one. That would tell me the tier can be a skill after all. I have not seen it, and the seeded control is the strongest evidence I could construct against my own position: I gave the voluntary channel every advantage I could think of, and it returned zero.

Close: If a Fact Must Survive, Deliver It

The wrong half I was solving, back at the top, was the store. I had built retrieval that understood code and a good place to keep what an agent learned, and I had left the most important step in the pipeline as an exercise for the model: notice you need this, then go and get it.

It does not notice. The capability is there, but noticing competes with the work and the work wins, every time, and it wins hardest exactly when the stakes are highest. Meanwhile the instruction file, which is the crudest channel in the whole stack and has no retrieval logic in it at all, wins on reliability because it never asks the model for anything. Everything I have built since has been an attempt to keep that property and drop the cost. That is the whole of what a cue is: an instruction file that is only present at the moments it is true.

So the practical version, if you are building on an agent right now. Take the facts your system must not lose and give each one a delivery rule instead of a storage location, expressed as something your harness can evaluate on its own. Make it as narrow as your event surface allows, and expect the event surface to be the binding constraint. Whatever refuses to be expressed that way stays a lookup, and that is fine, so long as you have stopped quietly assuming the lookup will happen.

The paper puts it more compactly than I have managed anywhere in twenty-seven minutes of this: the reliable memory channel for agents is the one the agent never has to think about.

Links and Further Reading

The paper and the runs

Saha, S. Delivery, Not Storage: Cue-Anchored Working Memory as a Harness Property for Coding Agents. arXiv:2607.20972, 23 July 2026. Every number in this post is from here.
Run archives for the adoption and compaction runs: protocol, graders, and per-run artifacts, including invalidated launches and their diagnoses.
vectr, the working-memory daemon used as the instrument, and the implementation the trigger examples are drawn from.

Harness behaviour

Anthropic. How Claude remembers your project. Documents instruction-file re-injection after compaction and path-scoped rules loading on touch.
Persistent Memory Across Context Compactions (59 compactions, built our own), a field report from a team that hit the same wall independently.
Triggered injection for auto-memory topic files, the upstream feature request the measurements motivated.

The cognitive science behind the two tiers

Berntsen, D. et al. Involuntary autobiographical memories and their relation to other forms of spontaneous thoughts. Phil. Trans. R. Soc. B.
Retrieval intentionality and forgetting: how retention time and cue distinctiveness affect involuntary and voluntary retrieval. Memory & Cognition.

Cue-driven retrieval in cognitive architectures

An Analysis and Comparison of ACT-R and Soar.
ACT-R, the cognitive architecture whose declarative memory model is the closest prior art to a cue-anchored tier.

Your MCP Tool Works. The Model Still Won't Call It.

Swapnanil Saha — Wed, 22 Jul 2026 15:36:06 +0000

I built a tool I was sure agents would use. It solved a real, felt pain — an AI coding agent losing everything it learned the moment its context filled up — and it was one clean install away from any editor that speaks the protocol. Then I ran the honest test: I connected it, gave an agent a real task, and counted the calls. The count was zero.

Not zero because the tool was broken. Every function was reachable, every response correct, the connection verified in the session's own startup log. Zero because the model, turn after turn, simply chose to do something else. That gap, between a tool that works and a tool that actually gets used, is the most under-measured number in the whole agent stack, and it's what this piece is about.

I'm going to show you the measurements: the run where an agent ignored a tool it was told about thirty times, the control where the answers were already sitting in the store and it never looked, and the counter-evidence where the exact same tool suddenly got used the instant the harness stopped asking and started delivering. Then I'll explain the mechanism, and give you a ladder you can climb from "polite invitation" to "the model has no choice."

Part 1 — The Unmeasured Multiplier

1. The Benchmark Nobody Runs

Every tool eval I've ever seen — mine included, for a while — measures the same thing: how good the tool is when it runs. Retrieval precision. Latency. Correctness on a held-out set. You tune, you publish a number, you feel good. And then that number gets multiplied by a second number nobody put on the slide: the fraction of the time the model actually decides to call the thing.

Write it out and the problem is obvious.

The value equation

Shipped value = retrieval quality × adoption × good use.

A tool that is 10× better at retrieval, multiplied by zero calls, ships zero value. Every benchmark you publish about your tool measures the first factor. The model decides the second. And — this is the part that took me a full quarter to accept — the default matters more than the quality.

Here's the trap. The first factor, retrieval quality, is where all the fun engineering lives: embeddings, rerankers, chunking, graph traversal. It's legible, it benchmarks cleanly, it feels like progress. The second factor is a behavioral property of a model you don't control, in a harness you don't own, and it doesn't show up in any dashboard unless you go looking. So nobody looks. They ship the quality improvement and quietly assume adoption is 100%.

It is not 100%. In the runs I'm about to walk through, it was closer to 0%. And you can only find that out one way — by measuring it, on purpose, before you spend three months polishing a factor that gets multiplied by nothing.

2. The Headline Zero

The instrument for all of this is a tool I wrote called vectr — a working-memory and code-search server that speaks MCP, the Model Context Protocol. I evaluated it in a controlled, multi-arm study on a naturalistic coding task, with the full protocol and every per-run artifact archived publicly so anyone can audit them. I care about the tool, but the number below is not about the tool. It's about what agents do.

One arm ran a 63-turn task with the MCP server connected and verified — all 10 tools visible in the session's initialization — and the workspace instruction file carrying 30 mentions of the tools and exactly when to use them. Thirty. Not a passing "you have some tools"; a document that repeatedly, specifically, named the tool and the moment to reach for it.

The result

The agent made 0 tool calls to the server. Zero, in 63 turns. A connected, correct, heavily-advertised tool, and not one call across an entire task.

Sit with how strange that is. If you had asked me to predict the number before the run, I'd have guessed something modest — maybe the agent leans on it a handful of times, maybe it forgets. I would not have guessed zero. Zero is not "under-used." Zero is the model treating a fully-wired capability as though it were not there.

A turn, here, is one cycle of the agent's loop — think, act, observe. Sixty-three of them went by. The agent read files, ran shell commands, grepped, wrote code. It reached for its defaults every single time. The one thing it never did was call the tool that had been connected specifically to help it, and described to it thirty separate times.

Before you reach for the usual explanation: this wasn't tool overload. The well-worn advice — "you've connected too many MCP servers, the model is drowning in tool schemas" — is real, but it doesn't fit here. This was one server, ten tools, cleanly connected, with nothing else competing for the model's attention. A single tidy toolset, and still nothing.

My first instinct was that I'd broken something. I re-checked the connection. Fine. I re-read the instructions. If anything they were too pushy. The tool wasn't the problem, and neither was the guidance. Something more structural was going on, and the only way to find it was to stop guessing and run the controls.

Part 2 — It Isn't a Guidance Problem

3. More Mentions, Same Zero

The obvious hypothesis after a zero is "the instructions weren't strong enough." So the natural fix is to turn up the guidance. I want to kill that hypothesis early, because it's where most teams sink their next month.

Across all five unseeded, tool-equipped runs of the matrix, voluntary memory writes came in at 0–1 per run. That's the whole distribution: some runs wrote nothing, the best wrote a single note. This was against 32 memory-guidance mentions spread across the workspace instruction files and a tool surface that was verified connected in every run. More words did not buy more calls.

And here's the detail that rules out the tidy "the agent is biased against third-party MCP tools" explanation. The harness has its own native memory feature — a built-in place the model is trained to write notes to itself, no MCP involved. In these runs, that native auto-memory directory was never created in any run either. The model didn't skip my tool in favor of the built-in one. It skipped voluntary memory of any kind.

The misdiagnosis that costs you a month

"Usage is low, so my instructions must be too weak" leads straight to writing more instructions. But the failure isn't dose. Zero native-memory writes alongside zero MCP writes tells you the model isn't rejecting your tool specifically — it's declining a whole category of voluntary behavior. No amount of louder prose changes the category.

This is the moment to separate two things that feel identical from the outside. There is the quality of your tool, and there is the model's standing disposition to reach for a non-default action unprompted. Guidance density pushes on the first — it makes the tool sound better, more relevant, more clearly the right choice. It does almost nothing to the second, because the second isn't a belief the model holds about your tool. It's a groove worn into the model by training. You can't argue a groove flat.

4. The Answers Were Already Inside

If louder guidance can't do it, maybe the problem is that the store starts empty — an agent won't consult a memory it has no reason to think contains anything. Fair. So I ran the control that removes that excuse entirely.

In the seeded-voluntary arm, the store was pre-seeded with four task-relevant notes — not filler, but real gotchas about the exact task the agent was about to attempt. The kind of thing that, if the agent read it, would save it a wrong turn. Same connected tools. Same guidance. The only change: the answers were already sitting inside, waiting to be recalled.

The strongest control

The agent made zero memory calls in 114 turns. One hundred and fourteen turns, four correct answers to its own problems within reach, and it never once looked.

This is the result that changed how I think about memory tools, so let me be precise about what it does and doesn't show. It does not show the notes were useless — they were exactly on point. It shows that relevance in the store is invisible to a model that never queries the store. Knowledge that isn't delivered might as well not exist. A perfectly-stocked library with a door the reader never opens is, functionally, an empty room.

The unopened fridge

Picture a housemate who orders takeout every night while the fridge is full of food you cooked and labeled for them. You could stock it better. You could tape a bigger note to the door. None of it works, because the failure isn't the contents of the fridge — it's that opening the fridge was never part of their routine. The only thing that ever works is putting a plate in front of them.

Storage without delivery is dead weight. That sentence sounds like a slogan, but it's the literal reading of a 114-turn run. If you're building a memory tier for agents, this is the load-bearing fact of your entire product: the value was never in holding the knowledge. It's in getting it in front of the model at the moment the model needs it — a moment the model, left to its own devices, will not reach for.

Part 3 — Why the Model Won't Call You

5. Training Priors and the Agency Tax

So why does a capable model, told thirty times, with the answers in hand, still not call? Two forces stack up, and once you see them the zeros stop being surprising.

The first is training priors. Agent models are trained, and then system-prompted, toward a specific default toolset: read the file, grep the repo, run the command. They're also trained to treat disk files as the canonical place memory lives. These aren't neutral capabilities the model weighs freshly each turn; they're the well-worn path. When the model needs to know something, "read a file" is the reflex, and reflexes fire before deliberation.

The second is what I've started calling the agency tax. Look at how the things that do reliably reach the model actually get there. A CLAUDE.md instruction file isn't something the model chooses to consult — the harness reads it and pastes it into context at startup, with zero model agency. The native memory index is loaded the same way. They "work" not because they're persuasive but because a mechanism outside the model puts them in front of it unconditionally.

A third-party MCP tool has none of that. To benefit from it, the model has to perform a fresh act of choice — notice the tool is relevant, decide to call it, form the arguments — and it has to do this at exactly the moments it's most loaded: mid-task, context filling, juggling the actual work. That's the tax. Every voluntary call costs a scarce unit of deliberate agency, spent against the pull of a reflex that's free.

The tax you pay in the moment you can least afford it

It's the gym membership you have but don't use. Not because you doubt exercise works — because using it demands a deliberate act at the exact moment your day is fullest. Meanwhile the couch requires no decision at all. Injected context is the trainer who shows up at your door and hands you your shoes. Same activity, but the choice has been made for you.

Put the two together and the zeros are almost overdetermined. The model's cheapest path is its trained default. Your tool sits behind a deliberate choice it must make while busy. Guidance tries to tilt that choice, but it's tilting against a slope that training carved and the harness reinforces for free on every other surface. No wonder "prefer these tools" reads, to the model, as a suggestion it's welcome to skip.

6. What the Docs Actually Say

You don't have to take my framing on faith — the harness vendors describe this split themselves, in their own documentation, in plain language. I pulled these quotes from the current Claude Code docs while writing this.

On what an instruction file even is, the memory documentation is blunt:

From the Claude Code memory docs

"Both are loaded at the start of every conversation. Claude treats them as context, not enforced configuration. To block an action regardless of what Claude decides, use a PreToolUse hook instead."

And, on why compliance varies: "CLAUDE.md content is delivered as a user message after the system prompt, not as part of the system prompt itself. Claude reads it and tries to follow it, but there's no guarantee of strict compliance, especially for vague or conflicting instructions."

Read that carefully. The canonical mechanism for guiding an agent is, by the vendor's own account, context, not enforced configuration, with no guarantee. That's not a bug to be fixed with better wording. It's the design. Instruction files shape behavior; they don't compel it.

And when the docs describe the thing that does compel, notice the contrast in verbs:

From the Claude Code hooks docs

"Hooks are user-defined shell commands that execute at specific points in Claude Code's lifecycle. They provide deterministic control over Claude Code's behavior, ensuring certain actions always happen rather than relying on the LLM to choose to run them."

"Ensuring certain actions always happen rather than relying on the LLM to choose." That single clause is the entire thesis of this piece, written by the people who build the harness. There's a category of thing the model is invited to do, and a category the harness guarantees. Your MCP tool, by default, lives in the first category. The features that never fail to reach the model all live in the second.

So this was never a marketing problem. You can't wordsmith your way from "invited" to "guaranteed." It's an architecture problem, and it reduces to one line I'd tattoo on every MCP builder: anything that depends on the model choosing to call you loses to anything the harness delivers unconditionally.

Version caveat, stated once

The measured runs used Claude-family agent models from mid-2026 in an agentic CLI harness. Adoption behavior is model- and version-specific, and training toward tool use is a moving target — a future model could reach for tools more readily. The mechanism (invitation vs. delivery) is structural; the exact zeros are a snapshot.

Part 4 — What Actually Moves Adoption

7. The Adoption Ladder

Here's the useful part. Adoption isn't binary — there's a ladder from weakest to strongest, and I've operated all four rungs on this same tool. Each one buys you more reliability and costs you more integration. The mistake is starting at the top and never climbing; the other mistake is thinking rung one is a real rung.

Rung 1 — Polite guidance. "Prefer these tools." This is the 30-mentions run. Measured effect: roughly zero. It is the most common thing MCP builders ship and the least effective. If your entire adoption strategy is prose in an instruction file, you are, empirically, shipping nothing.

Rung 2 — Prescriptive prohibition. Not "prefer X" but "use X for all code exploration; do not use file reads or grep for browsing." I found this early in the project: "prefer" is ignored, while "do not use the default" is what registers. Forbidding the reflex works better than praising the alternative, because it attacks the prior directly. But it's partial and fragile — it holds best in directly-driven prompts, and it re-litigates the fight against training priors on every single turn. Lean on it and you'll get compliance that quietly erodes as the task gets long.

Rung 3 — Response-packing. Make every tool response carry what the caller needs next. A search result that names the exact id to fetch for more. A write confirmation that shows, inline, how to recall what you just stored. It's deterministic and additive — you're not persuading the model, you're paving the road so the next call is the path of least resistance. This lowers the per-call agency tax meaningfully. Its limit: it can only help after the first call. It makes the second step cheap; it doesn't make the first step happen.

Rung 4 — Harness injection. Remove the choice. Don't ask the model to call you — have the harness deliver your content as context the model can't skip. This is the only rung that beats the priors, because it stops playing their game. It's also the rung that depends on your harness cooperating: Claude Code exposes hook surfaces you can inject through, but not every MCP host does, which is exactly why part of the fix belongs upstream.

8. Injection Begets Engagement

The best evidence that delivery is the lever isn't the zeros. It's what happened when I stopped inviting and started delivering. Same tool, same task family. The only change: the harness injected the relevant notes itself, at session start and per prompt, instead of hoping the model would go get them.

And the agents didn't just passively receive. They came alive.

In one run, the agent's first three tool calls were recalls of exactly the three injected note ids — ids it could not have known from anywhere except the injection that had just landed seconds earlier. It saw the delivered context, and immediately pulled the threads. Another run issued a semantic recall about 17 seconds in. A third produced a full memory-hygiene loop on its own: recall a note by id, forget a gotcha that its own change had just made obsolete, then remember a completion note for the next session.

The clincher

In a separate long-horizon probe, the injection-equipped agent voluntarily issued 11 remember calls and 9 recalls mid-run — against zero voluntary operations in the matched voluntary-only control. Same tool, same task. The difference was that one of them had been handed a plate.

Read the sequence carefully, because it inverts the intuition most people carry. You'd expect injection and voluntary use to be substitutes — if the harness is force-feeding context, why would the model bother calling the tool itself? The opposite happened. Delivery didn't replace engagement; it triggered it. Once the model had a delivered note in front of it, calling the tool became the obvious next move, and it kept calling. The plate on the table is what got the housemate to finally open the fridge for seconds.

This is worth stating flatly because it's the hopeful part: delivery is not a substitute for engagement; it is the cause of it. You are not choosing between "force it" and "let the model use it naturally." Forcing the first contact is how you unlock the natural use.

One more thing injection has to earn: it has to be safe. Content the model can't skip is exactly as dangerous as it is powerful — a mis-fired injection is noise the model can't ignore. So the trigger logic was audited. Across the gated runs, there were zero false-alarm injections across 40 and 35 audit-logged trigger evaluations, and the file-anchored gotcha fired on the first touch of its anchored file in both runs. Delivery only earns the right to be unconditional if it's also disciplined.

9. Match Delivery to the Stakes

"Just inject everything" is the wrong lesson. Injected context is a finite, expensive resource — every token you deliver unconditionally is a token the model can't spend on the task. The craft is matching the delivery mechanism to the stakes of the content. In a Claude Code-style harness, the hooks give you three natural tiers.

Content type	Stakes	Delivery mechanism
Never-deviate directive	A miss is unacceptable — "never push to main," "never delete prod data"	Unconditional injection at every SessionStart. No similarity threshold — a semantic near-miss on a hard rule is a catastrophe.
File-scoped gotcha	Only matters when a specific file is in play	Trigger on file touch via PreToolUse. Fires exactly when the anchored file is opened, and stays silent otherwise.
Episodic finding	Useful if relevant to this turn, harmless to skip	Ranked semantic recall per prompt via UserPromptSubmit — use the user's own prompt as the query, inject the top matches.

The mechanism has to fit the failure mode. A never-deviate rule cannot ride on semantic similarity, because the one time the retriever scores "never push to main" just below the cutoff is the one time it mattered most. That rule belongs in unconditional injection, full stop. An episodic finding, on the other hand, would be wasteful to inject every turn — ranked recall keyed to the prompt is exactly right, because a miss just means the model does what it would have done anyway.

Why "let the model search its memory" fails

One flat tool that says "the model can search its memory when it wants" collapses all three tiers into a single voluntary call — and then delivers none of them reliably. Standing rules become skippable. File-scoped caveats fire only if the model happens to query at the right moment. Episodic recall competes with the reflex to grep. The conflation is the bug. Three stakes levels need three delivery mechanisms.

This is the design principle the whole study points at: match the delivery mechanism to the stakes. It's more work than shipping one tool and a paragraph of guidance. It's also the difference between a memory layer that exists and one that fires.

Part 5 — For People Shipping MCP Servers

10. What To Do On Monday

If you have an MCP server in the wild or about to be, here's the short list I wish someone had handed me before I spent a quarter tuning a factor that was being multiplied by zero.

Measure adoption before quality. Connect your server, give the agent a real task (a naturalistic one, not "please use the tool"), and count your tool's calls. That's the whole test. The number may be zero. Far better to learn that in an afternoon than after a quarter of retrieval tuning that never reaches a user. This is the single highest-leverage hour you can spend on your tool, and almost nobody spends it.

Don't ship guidance prose; ship defaults. Prose in an instruction file is rung one, and rung one is empirically ~zero. If the harness exposes hooks or any injection surface, integrate there. Context beats invitation, every time, by construction. Your README's tone doesn't matter; your delivery mechanism does.

Design responses for the next call, not for completeness. Every response your tool returns should carry the next affordance — the id to fetch, the command to recall, the follow-up that's probably coming. You're not writing documentation, you're paving a road. Completeness is for humans reading logs; the model needs the next step made cheap.

If your tool is a memory tier, delivery is the product. The 114-turn control settles this. Storing knowledge is table stakes and, on its own, worth nothing to an agent. The product is getting the right note in front of the model at the right moment — which means your real engineering surface is triggers and injection, not storage and indexing. I develop this argument in full in the full write-up.

Push for first-class harness support for triggered delivery. Right now, injection lives in hooks you wire up yourself. It should be a first-class harness feature — memory files with rules-style paths: and events: frontmatter, default-off, so the harness delivers them on the right trigger without every builder reinventing it. That's the substance of an upstream feature request I filed, claude-code#78795, with a related field report in claude-code#34556. If this problem bites you too, that's the thread to pile onto.

The honest caveats

Sample sizes here are small — a controlled matrix over one task family. The claim is existence and mechanism ("adoption can be ~zero even with maximal affordance, and injection flips it"), not a universal adoption rate you can plug into a forecast. Behavior is model- and version-specific. And I'm the author of the tool under test, so the runs are self-run — which is exactly why the protocol, per-run artifacts, and graders are public. Don't trust the story; audit the runs.

The Multiplier, Again

I opened with a tool that got zero calls and a value equation with a middle term nobody measures. Those are the same story. The retrieval quality I'd been polishing was real; it just never got the chance to matter, because the factor next to it — adoption — was sitting near zero and quietly zeroing the product.

The fix wasn't a better tool or a louder pitch. It was accepting that a model reaching for its trained reflexes will not, on its own, climb over the agency tax to call you — not with thirty mentions, not with the answers already in the store. The things that reliably reach a model reach it because a mechanism outside the model puts them there. So you become that mechanism. You stop inviting and start delivering, and you match how forcefully you deliver to how much a miss would cost.

The next time you're about to publish a benchmark for your tool, run the other one first. Connect it, hand an agent a real task, and count. If the number is zero, no amount of quality was ever going to save you — and now you know precisely which factor to go fix.

Notes & Sources

Every number in this piece comes from the controlled runs, whose protocol and per-run artifacts are public. The vendor quotes were pulled from the live Claude Code documentation while writing.

The runs & the argument

Proactive-gate run archive — protocol, per-run artifacts, and graders for every number quoted here.
Delivery, not storage — the full write-up developing the memory-tier argument.
vectr — the MCP working-memory + code-search server used as the instrument. Local, MCP, no API key.

Harness documentation (verified live)

Claude Code — memory / CLAUDE.md — source of "context, not enforced configuration" and the compliance caveat.
Claude Code — hooks — source of "deterministic control … rather than relying on the LLM to choose."

Upstream

claude-code#78795 — feature request: triggered injection for auto-memory topic files (paths:/events: frontmatter, default-off).
claude-code#34556 — related field report.

Written by Swapnanil Saha. Originally published at swapnanilsaha.com.

Embedding Dilution: Why Semantic Code Search Misses the Answer

Swapnanil Saha — Mon, 20 Jul 2026 18:12:46 +0000

I had a query that should have been boring. "Get a single object from the database." In Django, that is QuerySet.get — the method whose entire job is to fetch exactly one row matching your lookup, or raise. Its docstring reads, almost verbatim: "Perform the query and return a single object matching the given keyword arguments." That is not a loose match to my query. It is nearly a paraphrase of it.

The chunk was indexed. I checked. The embedding was computed and stored like every other chunk in the corpus. And when I ran the search, the method was not in the top result, not at rank fifty — it was not in the top two hundred candidates at all. It never made it far enough into the pipeline to be judged. Two hundred other chunks, none of which described getting a single object from the database, beat it into the pool.

That failure bothered me enough to take apart the whole retrieval path and measure where the answer died. This post is the post-mortem. The system under test is vectr, a semantic code-search and working-memory tool I build; the corpus is Django, used purely as a public witness. But the mechanism I found is not specific to code, and it is not specific to my tool. It is a property of how a single embedding vector has to summarize a long, mixed passage — and it quietly limits recall in a lot of retrieval systems that look like they are working. If you have never read the embeddings foundations, my earlier complete guide to text embeddings and RAG is the primer; this is the failure that guide's happy path hides.

A note on terms. An embedding is a list of numbers (a vector) that captures a passage's meaning, so passages with similar meaning get similar numbers. A chunk is one indexed piece of the codebase — roughly one method plus a little context. A docstring is the documentation written inside a function. Cosine similarity measures the angle between two vectors: 1.0 is identical direction, 0 is unrelated.

Part 1 · The Miss

The Pipeline and the Query That Broke It

Before the failure makes sense, you need the shape of the pipeline it happened in, because the shape is where the whole story turns. At measurement time, vectr retrieved in the way most production semantic-search systems do — a two-stage funnel.

Stage one is hybrid retrieval. It runs two searches in parallel and merges them. One leg is dense retrieval: encode the query into a vector, encode every chunk into a vector, and rank chunks by cosine similarity — so it can match on meaning even with no shared words. The other leg is BM25, a keyword-scoring function that rewards exact term overlap. Together they produce a candidate pool of the top 200 chunks.

Stage two reranks that pool. A cross-encoder reranker — bge-reranker-base — reads the query alongside each of the 200 pool members and re-scores them properly, followed by a quality pass. That reranker is the smart part of the system. It is also the expensive part, which is exactly why it only ever sees 200 candidates instead of all 40,538.

The one number that decides everything. The reranker, the importance priors, the quality scores — every clever thing downstream operates only on the 200 chunks in the pool. A chunk that is not in the pool is invisible to all of it. So the first question for any retrieval miss is never "why did the reranker score it low." It is "was it even in the pool to be scored." Recall gates everything after it.

Now the pieces that matter for the failure. The dense embedder was snowflake-arctic-embed-m-v1.5 — a general text embedder, not a code-specialized one. Hold onto that; it is not the villain, but it shapes the numbers. The corpus was a Django checkout from June 2026: 4,129 files, 40,538 indexed chunks. And the chunk for QuerySet.get was, by any reasonable standard, ideal. It carried a class marker (QuerySet), the full method signature, and that near-perfect docstring — followed by roughly thirty lines of mechanical implementation.

That last clause is the whole problem in embryo. But to see why, we first have to pin down where the miss happened, because the fix depends entirely on that.

The Miss Is at Pool Entry, Not Ranking

My first instinct was the wrong one, and it is probably yours too: the reranker must have mis-scored it. Bump the reranker, add an importance prior for well-known symbols, tune the quality pass. Every one of those fixes operates on the pool. So I checked the pool directly, leg by leg, and the reranker turned out to be innocent — it never got the chance to be guilty.

The dense leg. For every natural-language phrasing I tried — "get a single object from the database," "fetch one row matching criteria," "retrieve a single record by lookup" — QuerySet.get was absent from the top 200 dense results. Not low-ranked. Absent. The only phrasing that got it into the dense pool at all was a deliberately ORM-flavored control, written in Django's own vocabulary, and even that reached only #123 of 200 — barely inside a pool it should have topped.

The keyword leg. BM25 was all over the place, which is its nature: it lives and dies on exact term overlap. Phrase the query as "return exactly one matching object or raise…" — words that literally appear near the method — and BM25 ranked it #1. Phrase it as "fetch one row matching criteria" and the same method fell to #127. Other phrasings missed entirely. BM25 wasn't a safety net; it was a coin whose bias depended on whether I happened to echo the source text.

Where the answer actually died. The reranker never saw QuerySet.get for the natural-language queries, because QuerySet.get was never in the 200 it was handed. This is the structural point the rest of the post builds on: if the right chunk never enters the pool, nothing downstream can save it. A brilliant reranker on an incomplete pool is a brilliant answer to the wrong question.

This is also the moment most retrieval dashboards lie to you by omission. They display the reranked top-k — the final, polished output — which looks fine because the reranker did a competent job on the pool it received. The miss is one layer up, invisible on that screen. You have to instrument pool entry itself to see it.

Trace the chunk's fate through the funnel (three real scenarios from this run):

Phrasing	Dense top-200	BM25 top-200	Fused pool	Outcome
A · "get a single object from the database"	absent (>200)	missed	not in pool	miss
B · ORM-vocabulary control	#123 of 200	—	enters (weak)	reaches reranker
C · "return exactly one matching object or raise…"	—	#1	dropped	miss

Scenario C is the case where BM25 ranked the target first, yet the dense-dominated fusion dropped it before the returned top-60 — more on that in Part 4. A dash means that leg's rank was not separately recorded for that phrasing.

Part 2 · The Cause

Dilution, Measured

So the dense leg failed to rank a chunk whose docstring paraphrases the query. Why? The lazy answer is "the embedder isn't good enough, throw a bigger model at it." That answer is wrong, and I can show it is wrong with one micro-experiment.

I took the exact same embedder and embedded two things. First, the full chunk: class marker, signature, perfect docstring, plus the ~30 lines of body — the combinator handling, the _chain() call, the select_for_update checks, the NotSupportedError raises. Second, just the signature and docstring alone — call it the "purpose-only" version, the part that says what this is for with none of the machinery that says how it does it. Then I measured cosine similarity from each version to four query phrasings.

Query	Full chunk	Purpose-only	Delta
get a single object from the database	0.601	0.706	+0.105
fetch one row matching criteria	0.511	0.606	+0.095
retrieve a single record by lookup	0.529	0.625	+0.096
return exactly one matching object or raise…	0.617	0.678	+0.061

Purpose-only is closer to the query on every single phrasing, by +0.06 to +0.10 cosine. Same embedder, same docstring, same query. The only thing I removed was the implementation body — and the chunk got measurably more relevant to the thing it does. The signal that answers the query was in the chunk the whole time. The body was burying it.

Why a longer chunk drifts away from its own purpose. An encoder turns a passage into one fixed-size vector. Cosine similarity then compares directions:
cos(q, d) = (q · d) / (‖q‖ · ‖d‖)
The catch is d. Whether the model builds it by literally averaging its token vectors (mean pooling) or by a summary token that attends across all tokens, the result is one point that must stand in for the whole passage. For the literal mean-pooling case it is just an average:
d ≈ (1/N) · Σ eᵢ
A CLS- or attention-pooled encoder weights the tokens unevenly instead of averaging them flat, but the consequence is the same. Add thirty lines of body and you pour in dozens of token vectors pointing toward "loop, chain, raise, check." They pull d toward the body's center of mass and away from the docstring's direction. The docstring's contribution does not vanish — it gets outvoted. That is dilution: not a missing signal, a drowned one.

Now the calibration that turns this from a curiosity into a recall failure. In this embedding space, for the "get a single object" query, the weakest chunk that made it into the 200-deep pool sat at a cosine of about 0.697. Purpose-only scored 0.706 — over the line, into the pool, in front of the reranker. The full chunk scored 0.601 — under the line, out of the pool, invisible. The entire difference between "the reranker gets a shot at the right answer" and "the right answer is never considered" is that +0.10 of cosine the body ate.

A caveat on that floor: 0.697 was calibrated on the first query only, and the pool floor is a per-query quantity — treat it as a reference for that query, not a universal threshold. The general lesson is the recovered delta, which is positive for all four.

The instinct this kills: "just enrich the input." The reflex when recall is bad is to feed the embedder more context — add the class body, the surrounding file, richer metadata. Here that makes it worse. The purpose signal is already present; enriching the chunk only adds more body tokens to average against it. You cannot fix a drowning by adding water. The problem is the pooling of a long, mixed chunk into one vector — so the fix has to change what gets pooled, not what gets added.

Phrasing Doesn't Rescue It — and the Symbol Index Proves Why

There is an obvious objection here: maybe I just phrased the queries badly. Maybe the right words would have pulled the chunk in. So I ran a 60-query sweep — 10 topics, 6 phrasings each — to give phrasing every chance to matter.

It didn't. Rephrasing shuffled which wrong answers came back; it did not surface the right ones. Ask for a "signal dispatcher implementation" and the top results were 1-to-6-line re-export stubs — the little shim modules that just re-expose a name — while the real Signal class, the thing that actually implements dispatch, was absent from the top three. Across the sweep, conceptual queries kept returning wrong or incomplete top-5 sets no matter how I said them. Phrasing is a knob on the query side. Dilution is a problem on the document side. Turning the query knob cannot un-average a document vector.

Then came the control that settled it. vectr also keeps a deterministic symbol graph: a plain lookup from a name to its definition site, no embeddings involved. For every canonical symbol that semantic search had just missed, the deterministic lookup resolved it exactly and instantly — Signal, BaseCache, Query, SQLCompiler, QuerySet.get, each to the correct file and line.

This is the control that localizes the bug. Same corpus, same index build. The symbol table resolves the target perfectly; semantic search cannot find it. That gap is not the parser's fault, not the chunker's fault, not a missing document. The index and the symbol table were correct. The failure is purely in the embedding and search layer. When two views of the same index disagree this cleanly, the broken one tells you exactly where to look.

One more result, and it is the one that made me stop trusting cold semantic search on its own. I ran a round of "famous symbols" — targets every Django developer knows by heart. Cold semantic search was roughly a coin flip even there. get_object_or_404, QuerySet.get, and reverse were all absent from the top-5; ForeignKey came in at #2 and Paginator at #4, each sitting behind look-alikes with more generic wording. If a system cannot reliably surface get_object_or_404, the failure is not exotic. It is the common case wearing a docstring.

Part 3 · A Trap in the Scores

Normalized Scores Lie About Confidence

This one is a short aside, but it burned me while I was debugging the above, so it earns its place. While hunting the dilution bug I ran control queries for concepts that do not exist in Django core — CORS handling, for instance, which Django leaves to middleware and third-party packages. A search for something absent should come back empty, or at least visibly unsure.

It came back with five hits scored between 0.77 and 1.0, looking every bit as confident as a real match. Nothing in the corpus answered the query, and the system reported near-certainty anyway.

The reason is a modeling choice that is easy to make and easy to forget. The displayed score was the reranker's output after per-query normalization — rescaled so the best result of this query becomes ≈1.0. That rescaling throws away the only thing you needed: how good the top match is in absolute terms.

The top score is always ≈1.0 by construction. A per-query-normalized score can't tell you "nothing here matches," because it is defined to make the best available result look like a perfect one — even when the best available result is garbage. The number describes rank within the query, not relevance to the world. If you surface it as confidence, your UI will radiate certainty at the exact moment it has found nothing.

The practical rule that fell out of this: never show a per-query-normalized score as if it were confidence. Keep a non-normalized signal alongside it — a raw cosine, or a BM25 floor — so the system retains an honest way to say "nothing here is actually close." Recall failures like the dilution one are already invisible enough; a score that reads 0.99 over an empty result set makes them worse, because it converts a silent miss into a confident wrong answer.

Part 4 · The Fix That Shipped

The Fix I Shipped: Dual-Vector Indexing

The measurement points at its own fix. If purpose-only embeddings score +0.06 to +0.10 higher — enough, in the calibrated case, to clear the pool floor — then the answer is not to throw away the body vector. It is to also keep a purpose vector, and let a query match whichever one fits it.

That is dual-vector indexing. At index time, store two vectors per symbol: a purpose vector built from the qualified signature and docstring with the body stripped out, alongside the existing full-body vector. At query time, retrieve over both, and blend or take the max of the two similarities for pool entry. Nothing else in the pipeline changes.

Analogy · The book and its spine. A full-body embedding is like shelving a book by blending every word in it into one average color. Two books with very different covers but similar bulk end up the same muddy shade, and you can't find either by its subject. The purpose vector is the printed spine: title and one-line description, nothing else. You keep the whole book on the shelf — you just also write a legible spine, so someone looking for the subject can find it without reading all 300 pages first. Dual-vector indexing shelves every symbol with both: the full text, and a spine.

What I like about this shape is that it does not privilege one kind of query. Intent-shaped queries — "get a single object from the database" — land on the purpose vector, where the docstring is undiluted. Implementation-detail queries — "where is select_for_update checked" — still land on the body vector, because that string only exists in the body. You are not trading one failure mode for its mirror image; you are giving each query the surface it needs.

Two properties make it safe to apply blindly across a whole corpus:

Undocumented symbols degrade gracefully. No docstring? The purpose vector is just the qualified signature. It never gets worse than the name itself, and the name is often enough.
It is a uniform structural transform. The same body-stripping rule applies to every symbol, at index time, with no query-side special-casing — no keyword lists, no "if the query looks conceptual, reroute it." Query-side heuristics are the thing I have spent a long time deleting from this system; they are brittle, they compound, and they never generalize. A transform on the index side has none of those failure modes.

It is not free, though, and I would rather name the costs than let you find them. Storing a second vector per symbol roughly doubles the number of vectors in the index — reason enough to confirm dilution is actually your problem before you spend on it. And it does not stand alone: dual-vector composes with structural ranking signals like symbol importance rather than replacing them. A diluted docstring and an under-weighted call graph are different failure classes; fixing one leaves the other exactly where it was.

The honest sequence, stated plainly: I measured the cause first, then shipped the fix. The spike proved the mechanism — purpose-only embeddings clear the pool floor where full-chunk embeddings do not — and that measurement, not a hunch, is why dual-vector indexing shipped in vectr v1.0.0 on 8 July 2026. If someone tells you a retrieval change "should work," ask them for the cosine table. This one had one before a line of the fix was written.

The fusion bug hiding underneath. While validating the direction I tripped over a second, separate problem worth its own paragraph, because it will bite anyone running hybrid retrieval. Remember that BM25 ranked the target #1 for one phrasing. You would assume a #1 in either leg guarantees pool entry. It did not: the fused final top-60 did not contain the target even though BM25 had ranked it first. The fusion was dense-dominated, and the dense leg's absence outvoted the keyword leg's #1.

Check that your fusion can't discard a leg's #1. Hybrid retrieval is supposed to be a safety net: if one leg misses, the other catches. That promise only holds if your fusion actually lets a strong single-leg result survive. A dense-dominated blend can throw away the exact result BM25 nailed. Before you trust hybrid search, feed it a query where you know one leg ranks the answer #1, and confirm the answer is still in the fused output. Mine wasn't.

The Same Dilution Shows Up Far Beyond Code

I found this in code search, but nothing about the mechanism is about code. Dilution appears in any corpus where a document mixes "what this is for" with "how it works" or with plain boilerplate. The purpose is a small fraction of the tokens; the pooled vector drifts toward the bulk; a query written in terms of purpose lands short.

You have almost certainly hit it without naming it:

API reference pages where a one-line summary sits on top of exhaustive parameter tables and examples. Search for what the endpoint does and the parameter soup dominates the vector.
Legal clauses buried inside pages of recitals and boilerplate. The operative sentence is three lines; the surrounding scaffolding is three hundred.
Product descriptions embedded in spec sheets, where the one line a buyer would search for is outweighed by dimensions, SKUs, and compliance notices.

The two mitigations generalize as cleanly as the problem does. First, embed a purpose or summary field separately from the full text, and retrieve over both — the dual-vector idea, minus the word "symbol." A short, curated summary vector per document is often the single highest-leverage change you can make to recall, precisely because it is immune to dilution by construction. Second, and this is the cheaper habit to build: audit recall at the pool level, not just the final ranking. Most RAG dashboards show you the reranked top-k and nothing else, which means a pool-entry miss is completely invisible on the screen you are staring at. The failure that started this whole post would never have shown up on a top-k view. I only found it because I went looking one layer up.

Choosing the embedder is a real lever — measure it, don't assume it. These deltas came from one general text embedder on one corpus. A code-specialized model would move the numbers; the CoIR benchmark evaluates nine retrieval models across ten code datasets and eight tasks and finds even state-of-the-art systems struggle with code retrieval, which is exactly why the embedder is not a detail. But a better embedder does not repeal dilution — it raises the whole curve, floor included, and a long mixed chunk still averages its purpose away. Dual-vector composes with a better model; it does not compete with one.

Part 5 · Takeaways

What To Actually Do With This

Go back to the opening. A function whose docstring paraphrased the query ranked below two hundred chunks that didn't. You now know the chain underneath that sentence. The docstring's signal was real and present. The thirty lines of body around it pulled the pooled vector away — enough to cost about a tenth of a cosine point. That tenth was the difference between clearing the pool floor and never entering the pool at all — between reaching the reranker and never being considered. The reranker never failed, because the reranker never saw it. And a normalized score would have happily reported confidence over whatever wrong answers did make the pool.

If you build retrieval, here is the short version to carry out of this:

Measure recall at pool entry, not at the reranked top-k. The reranker can only be as good as its pool. The miss that matters most is the one that never reaches the screen you monitor.
When recall is bad, test purpose-only against full-chunk. Embed a summary or signature alone, measure its cosine to the query, and compare. A large positive delta names your problem: dilution, not a weak model.
Index a separate purpose vector, and retrieve over both. Keep the full text for detail queries; add a body-stripped summary vector for intent queries. It is a structural transform on the index, so it needs no query-side heuristics to work.
Confirm your fusion can't drop a leg's #1, and never surface a per-query-normalized score as confidence. Keep a raw, absolute signal for an honest no-match.

The dual-vector fix shipped in vectr v1.0.0 on the strength of the cosine table above rather than a hunch. The boring query that started this — get a single object from the database — is exactly the case it was built to catch: the method whose docstring says precisely that, given a surface where its own body can no longer outvote it. The signal was never missing. It just needed somewhere to be read on its own.

Links & Further Reading

Every external claim in this post was confirmed against the source it points to; where a source did not confirm a specific number, that number is not stated here.

vectr — the semantic code-search and working-memory tool used as the system under test, and the instrument that produced these measurements.
The Complete Guide to Text Embeddings, Vector Databases & LLMs — the primer this post assumes: tokenization, pooling, cosine similarity, and how a RAG pipeline fits together.
CoIR: A Comprehensive Benchmark for Code Information Retrieval Models — a benchmark of ten code datasets across eight retrieval tasks and seven domains; it evaluates nine retrieval models and finds significant difficulty with code retrieval even for state-of-the-art systems. arXiv:2407.02883
Structural Code Search using Natural Language Queries — reports that a natural-language-driven structural search outperforms baselines based on semantic code search by up to 57% F1; embeddings alone under-serve structural queries. arXiv:2507.02107

Numbers here are from one embedder (arctic-embed-m-v1.5) on one corpus (Django, June 2026 checkout). The deltas are model-specific; the mechanism is general. Cosine thresholds like the ~0.697 pool floor are corpus- and index-specific calibration points, not universal constants.

The Four Families of Context Relief for LLM Coding Agents

Swapnanil Saha — Fri, 17 Jul 2026 13:50:58 +0000

Run a coding agent on anything bigger than a toy repo and you hit the same wall. The context window fills up. Not with the answer — with the search for the answer. Twelve file reads, four grep results, a stack trace, the output of a test run that failed for an unrelated reason. By the time the agent is ready to write the fix, half its working memory is archaeology it will never look at again.

I've spent the last few months building a semantic-search-plus-working-memory MCP server (I'll call it vectr throughout — it's the running example, not the point of the post), and the single most clarifying thing I did early on was stop treating "the context is full" as one problem. It's four problems wearing a trench coat. Each has its own mechanism, its own cost, its own failure mode, and — this is the part people miss — they only work when you compose them correctly. Get the composition wrong and you don't get relief; you get a subtle new class of bug where the agent confidently reasons over information it no longer has.

So here's the map I wish someone had handed me. Four families of context relief: what each one actually buys you, where each one bites, and how they fit together.

A few definitions first, because the jargon is dense. A token is the unit an LLM reads and bills by — roughly three-quarters of a word. The context window is the fixed number of tokens the model can attend to at once (a million, on the current Claude models). Prompt caching lets you pay a reduced rate to re-send an identical prefix instead of reprocessing it. And MCP (Model Context Protocol) is the standard interface an agent uses to call external tools. Keep those four in your head and the rest follows.

Family	What it does	Direction	Fails when…
1 · Eviction	Delete stale context, leave a placeholder	Removes	You threw away what you can't cheaply restore
2 · Offload & recall	Write findings to a durable store, fetch on demand	Restores	Recall isn't automatic — the model forgets to look
3 · Retrieval	Fetch the exact function, not the whole file	Restores	Used on structural questions (call graphs)
4 · Subagents	Burn messy work in a separate window	Removes	No shared memory — children re-derive everything

Family 1 — Eviction: throw it away, but keep a receipt

Eviction is the most literal answer to a full context: delete the stale stuff. The tool result from twenty turns ago, the file you read and already edited, the thinking block from a reasoning step that's now resolved — drop it out of the live window so the model stops paying to carry it.

The harness can do this for you, and increasingly it does. Claude Code calls this compaction: as a session approaches its context limit, it clears the oldest tool outputs first and only summarizes the rest of the conversation if that isn't enough on its own. Recent tool results stay inline so you can keep reasoning over them; older ones get cleared first.

At the API level there's a more configurable version. Anthropic's context editing feature exposes a strategy with the delightfully machine-generated name clear_tool_uses_20250919. You turn it on with a beta header (anthropic-beta: context-management-2025-06-27) and it watches your accumulating tool results. Once input tokens cross a threshold — the default trigger is 100,000 input tokens — it clears the oldest tool results in chronological order, keeping the most recent few (keep defaults to 3 tool uses).

Here's the detail that matters more than any of the parameters: each cleared result is replaced with placeholder text so the model knows it was removed. The agent doesn't silently lose a tool result and then hallucinate what was in it. It sees a tombstone — "this tool result was cleared" — which is a very different thing from a gap.

The config below overrides the defaults to make the behaviour easy to see — I've set trigger to 30,000 tokens rather than the stock 100,000 so it fires early, and pinned clear_at_least so each pass removes a real chunk. In production you'd leave trigger higher.

context_management={
    "edits": [
        {
            "type": "clear_tool_uses_20250919",
            "trigger": {"type": "input_tokens", "value": 30000},
            "keep": {"type": "tool_uses", "value": 3},
            "clear_at_least": {"type": "input_tokens", "value": 5000},
            "exclude_tools": ["web_search"],
        }
    ]
}

What it saves

Straightforwardly, input tokens — though the headline figure Anthropic has published for this feature is a task-performance number, not a token count: on an internal agentic-search evaluation, context editing alone lifted performance 29% over baseline, rising to 39% when paired with a memory tool. Hold onto that second number. It's the whole thesis of this post hiding in a benchmark.

What it costs

Two things, and the second is non-obvious. The first is the risk that you evict something the model actually needed — mitigated by the placeholder tombstone and by exclude_tools, which lets you mark, say, your search tool's results as never-clearable. The second cost is about prompt caching, and it's where a lot of naive eviction setups quietly lose money.

The cache-invalidation math. Prompt caching bills a cached prefix at 0.1× the base input rate on a read, but a cache write costs 1.25× the base rate for the default 5-minute TTL (2× for the 1-hour TTL). The catch is invalidation: the cache key is a cumulative hash of everything up to and including your cache breakpoint, so changing any block at or before the breakpoint produces a different hash and a full cache miss.

Read those two facts together and the tension jumps out. Eviction edits the middle of your conversation. Every time it fires, it changes the prefix, which invalidates the cache from that point forward, which means your next request pays the 1.25× write cost to re-cache the new prefix. Evict a little bit, often, and you can spend more on cache churn than you saved on the evicted tokens. This is exactly why the clear_at_least parameter exists: it forces each clearing pass to remove a worthwhile chunk of tokens so the cache invalidation is amortized against a real saving, not a rounding error. If you take one operational lesson from this whole family, make it that one — evict in big, infrequent passes, never in a trickle.

When it fails

Eviction fails the moment the model needs something you threw away and can't cheaply get it back. A tombstone that says "tool result cleared" is honest, but honesty doesn't reconstruct the file. If the only copy of that information lived in the evicted tool result, the agent is now stuck: it has to re-run the tool, re-read the file, re-derive the thing. You've converted a token cost into a latency-and-tool-call cost — and if the underlying state has changed in the meantime, possibly into a correctness bug.

The governing rule of eviction. You can only safely evict what you can cheaply restore. Eviction on its own is not a memory strategy — it's a bet that restoration is cheap. That bet is only good if something else in your system guarantees it. Which is the entire reason the other three families exist.

Family 2 — Offload & recall: write it down where it survives

If eviction is "throw it away and hope you don't need it," offload-and-recall is "write it down somewhere durable before you throw it away, and fetch it back on demand." The agent, mid-session, notices it has learned something worth keeping — a function signature, a gotcha, a decision, a partial result — and commits it to an external store. Later, instead of carrying that finding in the context window the whole time, it recalls it in a single cheap call exactly when it's relevant.

This is the working-memory pattern, and it's the half of vectr I care about most. When the agent discovers that, say, a workspace lock is acquired at resolver.rs:214 and released on scope exit, it doesn't keep that fact parked in context for forty turns. It stores a note. The note sits in a local store keyed to the workspace, and a recall call pulls it back — in my case in under 50 milliseconds — whenever the agent's current task touches locking.

The reason this is a distinct family, and not just "eviction with extra steps," is what it survives. A finding in the live context window dies three deaths. It costs tokens the entire time it sits there. It gets mangled or dropped when the conversation is compacted into a summary — compaction preserves the gist and loses the exact line number. And it vanishes completely when the session ends. A note in an external store survives all three. It's there after /compact. It's there in tomorrow's session. It costs nothing until you ask for it.

Anthropic's own memory tool works on this principle, and it's the reason for that 39%-versus-29% gap I told you to hold onto. Context editing alone improves performance 29% over baseline on Anthropic's internal agentic-search evaluation. Context editing plus a memory tool gets you 39% — because the agent writes the important bits to memory before the eviction pass clears them, so clearing becomes safe instead of lossy. That extra ten points isn't a second independent optimization stacked on the first. It's the same optimization made safe to run harder, because family 2 backs it up.

What it saves, what it costs

It saves the standing token cost of carrying a finding you only need occasionally. And, less measurably but more importantly, it saves re-derivation — the agent doesn't have to re-read the file and re-reason to the same conclusion next time.

Against that, three real costs. The agent has to decide what's worth remembering, which is a judgment call it will sometimes get wrong — store noise and your recall gets diluted. The recall has to actually be relevant when it fires, which is a retrieval-quality problem in miniature. And there's a token cost to the recall itself, a fact I had to make peace with: store terse one-line notes and recall is cheap but thin; store full code blocks and recall is rich but heavier. There's no free lunch. There's a dial.

When it fails

It fails when recall isn't automatic. If your architecture depends on the model choosing, of its own accord, to call the recall tool at the right moment, it will frequently just… not. The model has no reliable sense of what it stored three sessions ago. The fix is to stop relying on the model's initiative and inject the relevant notes into context deterministically — which, in Claude Code, means hooks, and which is a whole post of its own. The short version: an offload store the agent forgets to read is a filing cabinet in a locked room.

The note-taking engineer. Working memory is the difference between an engineer who takes notes and one who doesn't. The note-taker doesn't hold the whole system in their head at once — they hold a pointer to where they wrote it down, and the act of writing it down is cheap insurance against the cost of re-discovering it. The catch is the same for both: a note you never look at again is just slower forgetting.

Family 3 — Retrieval over stuffing: fetch the 40 lines, not the file

The third family attacks a different waste. The first two are about getting rid of information you already loaded. This one is about never over-loading it in the first place.

The default way an agent explores an unfamiliar codebase is grep-and-read. Grep for a likely keyword, get forty hits, read the six files that look plausible, discard five of them. Every one of those reads lands the entire file in context — a 400-line module of which the agent needed one function. The signal-to-noise ratio is brutal, and unlike a human skimming, the model pays full token price for every line whether it was useful or not.

Retrieval-over-stuffing replaces the blunt read with a targeted fetch. Instead of loading whole files and letting the model sift, you run a ranked retrieval — semantic search over the codebase, ideally chunked at function and class boundaries so each result is a self-contained unit of meaning — and hand back the forty lines that actually match the query. "JWT validation logic" returns the verify_token function directly, even though neither word appears in it, and it returns that function, not the 400-line file it lives in.

This is the search half of vectr, and the payoff on unfamiliar code is large: on a big Java codebase in my own benchmarks, ranked retrieval cut the read-and-grep calls before the first edit by roughly three-quarters compared to the grep-and-read baseline. The mechanism is boring — embeddings plus a keyword index, merged — but the discipline is the point: the unit you put in context should be the unit of meaning, not the unit of storage. A file is a storage unit. A function is a meaning unit.

What it saves, what it costs

It saves the bulk of exploratory token spend, and the turns that go with it. A search that returns the right function in one call replaces a grep-plus-four-reads sequence. In exchange, you need an index — which means an indexing step and the machinery to keep it fresh as files change — and retrieval quality becomes a first-class concern.

Confident-wrong retrieval is worse than a miss. A search that returns the wrong forty lines is more dangerous than a grep that returns nothing, because the agent trusts it more. I've watched an agent build a wrong mental model off a top-ranked result that was subtly off-topic, then reason confidently from that bad premise for a dozen turns. An honest empty result would have sent it looking again; a plausible wrong one didn't.

When it fails

It fails on questions retrieval is the wrong tool for. "Who calls this function?" is not a similarity question — the callers don't contain the callee's body, they contain a reference to it by name. That's a graph traversal, not a search. Reach for semantic retrieval there and you'll get plausible-looking garbage. Part of doing this family well is knowing which questions are retrieval questions (concepts, patterns, "how does X work") and which are structural ones (definitions, call graphs) that want an exact lookup instead.

Family 4 — Subagent isolation: burn the tokens in someone else's window

The fourth family is the cleverest and the easiest to get subtly wrong. The idea: when a subtask is going to generate a pile of context you'll never reference again — a research spike, a log-diving expedition, a broad search — you don't do it in your main conversation. You spawn a subagent, let it do the messy work in its own context window, and take back only the distilled answer.

Claude Code's subagents work exactly this way. Each one runs in its own context window with a custom system prompt, does its work independently, and returns only the result — the docs frame it as keeping exploration and implementation out of your main conversation. The parent agent spends, say, 800 tokens receiving a clean summary of an investigation that cost the subagent 40,000 tokens of reading and reasoning. Those 40,000 tokens are burned in a window that gets discarded. The parent's context stays clean.

There's a nice secondary benefit. Because a subagent has its own tool permissions and its own system prompt, you can also use it to constrain work — a read-only research agent that literally cannot write files — and to route cheap work to a cheaper, faster model. Context isolation and cost control fall out of the same mechanism.

What it saves, what it costs

It saves the largest single chunk of exploratory context there is. A well-scoped subagent is the difference between your main window holding a conclusion and holding the entire messy derivation of that conclusion. The cost is a framing-and-parsing tax at the boundary: you have to specify the subtask well enough that the subagent can run without hand-holding, and you have to trust the summary it returns without seeing its work. If the summary is lossy in exactly the way that matters, the parent proceeds on a bad abstraction — and it can't tell, because the detail that would have flagged the problem got left behind in the discarded window.

When it fails

Here's the failure mode nobody warns you about. Subagent isolation with no shared memory means every subagent starts cold. It re-derives context the parent already had and the last subagent already found. Spawn three subagents to investigate three corners of the same system and, without a shared store between them, each one re-reads the same core files, re-learns the same architecture, and re-discovers the same gotcha — three times, in three separate windows, at full price each. You've isolated the context so well that you've also isolated the learning. The isolation that saves the parent's window quietly taxes every child.

Isolation without a shared bus is a false economy. Subagent isolation without a shared memory store is a false economy at scale. You save the parent's context by making the children re-derive everything from scratch. The fix is to give the subagents the same durable store from family 2 — so the first subagent's findings are recalled by the next instead of rediscovered. Isolation controls what flows up; shared memory controls what flows sideways. You want both.

The families compose — that's the whole point

I've been dropping the composition hints deliberately, so let me make them explicit, because treating these four as a menu you pick one item from is the mistake I most want to talk you out of.

Eviction (1) is only safe on top of offload (2) or retrieval (3). This is the load-bearing relationship. You can only throw information away cheaply if you can get it back cheaply — and "getting it back cheaply" is precisely what families 2 and 3 provide. Evict a tool result whose contents you already wrote to a memory note: safe, because recall restores it. Evict a file you can re-fetch with one targeted search: safe, because retrieval restores it. Evict something that exists nowhere else and you've just planted a bug that will surface three turns later as confident nonsense. The 29% → 39% jump from adding a memory tool to context editing is this relationship, quantified — the memory tool is what makes the eviction safe to run harder.

Retrieval (3) keeps the working set small enough that eviction (1) rarely has to fire. If you never stuffed the whole file in, there's less to evict later. The two attack the same waste from opposite ends — one at load time, one at cleanup time — and a system with good retrieval needs less aggressive eviction.

Subagents (4) need a shared store (2) or they re-derive context. Covered above, but it's the composition people skip most often, because subagents feel self-contained. They're self-contained in their context, not in their knowledge. Wire them to the same working-memory store and the isolation stops being a re-derivation tax.

The unifying idea is almost embarrassingly simple once you see it: cheap restoration is the license to be aggressive about relief. Every family is either a way to remove context (1, 4) or a way to make removal safe by guaranteeing you can get the important parts back (2, 3). Build only the removal half and you get an agent that forgets things it needed. Build only the restoration half and you get an agent that never frees anything and grinds to a halt at the context limit. You need the pair.

This is also why I stopped thinking of vectr as "a search tool" or "a memory tool." It's families 2 and 3 in one MCP server, deliberately, because on their own each is half a solution. Search without memory re-explores every session. Memory without search has nothing good to store. And both of them exist, in the end, to make the harness's eviction — family 1, which I don't even own — safe to run.

A short field guide

If you operate a coding agent and want to actually apply this, here's the compressed version I'd give a colleague over coffee.

Start with retrieval (3), because it's the one that prevents the mess instead of cleaning it up, and it pays off immediately on any codebase you don't have memorized. Add offload-and-recall (2) next, and make the recall automatic rather than something the model has to remember to do — a store the agent forgets to read is worthless. Let the harness handle eviction (1), but check that it's evicting in big infrequent passes (mind the cache-invalidation math) and that everything it evicts is backed by 2 or 3. Reach for subagent isolation (4) on genuinely large exploratory subtasks, and if you use more than one subagent on related work, give them a shared memory bus or accept that each is paying full freight to learn what the last one already knew.

None of these is exotic. The compaction and context-editing pieces ship in the tools already. The retrieval and memory pieces are a weekend to prototype — I wrote up the honest numbers on how far mine actually got if you want the unvarnished version. What's rare is treating them as one system with a single governing rule — restore-ability licenses removal — instead of four disconnected tricks. Get the rule right and the context window stops being the thing you fight and starts being the thing you manage.

Four problems in a trench coat, then — not one. And once you've split them apart, the thing that surprised me is how little the individual tricks matter next to the relationship between them. Any single family, run on its own, either forgets something it needed or refuses to let go of anything. What actually works is the pair: a way to remove context sitting on top of a guarantee that you can get the important parts back. Cheap restoration is what buys you the right to be ruthless. Wire that in and the context window quietly changes from the wall you keep hitting into a budget you spend on purpose.

Sources

Claude Code — Context window and compaction. https://code.claude.com/docs/en/context-window (accessed 2026-07-07)
Anthropic — Context editing (clear_tool_uses_20250919, clear_at_least, keep, trigger, exclude_tools). https://platform.claude.com/docs/en/build-with-claude/context-editing (accessed 2026-07-07)
Anthropic — Managing context on the Claude Developer Platform (29% / 39% performance figures for context editing alone vs. context editing plus the memory tool). https://www.anthropic.com/news/context-management (accessed 2026-07-07)
Anthropic — Prompt caching (5-minute / 1-hour TTL, 1.25× / 2× write, 0.1× read, cumulative-hash invalidation). https://platform.claude.com/docs/en/build-with-claude/prompt-caching (accessed 2026-07-07)
Claude Code — Subagents (per-subagent context window, returns only the summary). https://code.claude.com/docs/en/sub-agents (accessed 2026-07-07)

Claude Code Hooks: A Practical Deep-Dive on Deterministic Agent Behavior

Swapnanil Saha — Sun, 12 Jul 2026 22:36:47 +0000

Here's a thing that took me embarrassingly long to accept about coding agents: you cannot instruct your way to reliability.

I had a working-memory system — a semantic-search-plus-notes MCP (Model Context Protocol) server I've been building, and it's the case study for this whole post — and it worked beautifully in demos. The agent would discover something, store a note, recall it later, save itself a re-read. Then I'd watch a real session and the agent would just... not recall. It had notes sitting right there, one tool call away, verbatim, and it would instead re-read the same file it had already read two sessions ago, because nothing made it check. My CLAUDE.md said "call recall at the start of every task." The model read that instruction and ignored it, the way it ignores roughly anything that competes with the task actually in front of it.

The lesson generalizes past my project. Any behavior you need to happen every single time — inject context, run a linter, block a dangerous command, snapshot state before it's destroyed — cannot depend on the model deciding to do it. The model is a probabilistic thing optimizing for the current turn. You need something outside the model, in the harness, that fires deterministically. In Claude Code, that thing is hooks.

This is the practical guide I wanted when I started: what the events are and what each is genuinely good for, the exact configuration and I/O contract (which is fiddlier than the docs make it look), and then a real production hook pipeline — mine — walked through end to end, including the design calls and the parts that bit me.

The problem hooks actually solve

Before the plumbing, the point. There is a whole class of things you want an agent to do that instructions are simply the wrong tool for. Not because the instruction is badly worded — because instructions target the model, and the model is the part of the system you don't control.

Think about what "the model complies with an instruction" actually means. On any given turn there's some probability the behavior happens, and that probability is well short of 1. It drops when the task gets absorbing, when the context is long, when an unusual prompt pulls attention elsewhere. That's fine for a preference — "prefer functional style," "keep commits small." It is a disaster for a guarantee. If the only thing standing between your agent and an rm -rf on the wrong directory is a politely worded line in a config file, you don't have a control. You have a hope.

Hooks move the decision out of the model and into the harness. The harness is deterministic: it runs code on a schedule, whether or not the model would have thought to. That single relocation — from "the model should" to "the harness will" — is the entire idea, and everything below is mechanics in service of it.

The core distinction. CLAUDE.md is where you put things the model should tend to do. Hooks are where you put things that must deterministically happen. Confusing the two — trying to instruction-engineer a guarantee — is how you end up with a system that works in the demo and flakes in production.

What a hook actually is

A hook is a shell command — or, increasingly, an HTTP call or an MCP tool invocation — that Claude Code runs automatically when a specific event fires in the session lifecycle. The event hands your command a JSON blob on stdin describing what's happening. Your command does whatever it wants and communicates back through two channels: its exit code and its stdout. That's the entire model. It's Unix-plumbing simple, which is exactly why it's reliable — there's no LLM in the loop deciding whether to honor it.

The events cover the session from birth to death. When I first wrote my pipeline the list was short; by mid-2026 it has grown considerably — the reference now documents around thirty event types spanning session lifecycle, per-turn, per-tool-call, permissions, subagents, worktrees, and MCP elicitation. Most of them you'll never touch. The workhorses — the ones worth learning cold — are these:

Event	Fires	What it's for
`SessionStart`	Session begins or resumes (matchers: `startup`, `resume`, `clear`, `compact`)	Inject state the agent needs before turn 1 — branch info, environment, recalled memory
`UserPromptSubmit`	Before Claude processes each user prompt	Inject per-turn context keyed to what the user just asked; can also block the prompt
`PreToolUse`	Before a tool call runs (matches on tool name)	Block dangerous calls, rewrite arguments, or surface a warning tied to the specific action
`PostToolUse`	After a tool call succeeds	React to results, replace tool output, add follow-up context
`PreCompact`	Before context compaction (matchers: `manual`, `auto`)	Persist anything that's about to be summarized away
`Stop` / `SubagentStop`	When the agent (or a subagent) finishes a turn	Enforce "you're not done yet" — block the stop and send it back to work
`SessionEnd`	Session terminates	Cleanup, flush, teardown
`Notification`	Claude Code emits a notification (permission prompt, idle, etc.)	Route notifications to your own channels

The mental split that helps: session-scoped events (SessionStart, SessionEnd) bracket the whole thing; per-turn events (UserPromptSubmit, Stop) fire once per user exchange; per-tool events (PreToolUse, PostToolUse) fire around individual tool calls, potentially dozens of times a turn. Match the cadence of your hook to the cadence of the thing it's reacting to, or you'll either miss events or fire far too often.

The configuration surface

Hooks live in settings.json. There are three tiers, and the tier decides who the hook applies to and whether it's shared:

~/.claude/settings.json — all your projects, never checked in.
.claude/settings.json — one project, checked in and shared with the team.
.claude/settings.local.json — one project, gitignored, personal.

The shape is nested and, honestly, a little awkward until it clicks. Under a top-level hooks key, each event name maps to a list of groups. Each group has an optional matcher and a list of hooks to run:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/guard.sh",
            "timeout": 30
          }
        ]
      }
    ]
  }
}

The matcher is the filter. For tool events it matches against the tool name — "Bash", "Edit|Write" for either, or a regex like "mcp__memory__.*" for a whole MCP server's tools. For non-tool events it matches against the event's reason: SessionStart takes startup|resume|clear|compact, PreCompact takes manual|auto. Omit the matcher (or use "*") to fire on everything.

The type used to be implicitly "command." It's now explicit and there are several — command (shell), http (POST to a URL), mcp_tool (invoke an already-connected MCP tool), and two LLM-in-the-loop types (prompt, and the experimental agent) that let a hook ask a model to make a yes/no call. For anything latency-sensitive you want command, because it's a local process with no network round trip. The useful knobs on a command hook are timeout (seconds; the default is generous but UserPromptSubmit is capped lower because it's on the critical path of every turn), and the ${CLAUDE_PROJECT_DIR} placeholder so your command path survives the user's working directory changing.

The I/O contract, which is where people trip

This is the part the quickstart glosses and the part that determines whether your hook works. A hook talks back through exit code and stdout, and the two are read differently depending on the code.

Exit code 0 — success. Claude Code parses stdout looking for JSON. If it's JSON, the fields are honored; if it's not, no decision is taken and the session proceeds normally. (Two events, SessionStart and UserPromptSubmit, are more generous: they also fold plain, non-JSON stdout straight into the context. For every other event, if you want to inject something you emit the JSON form.) This is the channel you use to inject context.

Exit code 2 — blocking error. stdout's JSON is ignored; instead, stderr is read as an error message, and for blockable events (PreToolUse, UserPromptSubmit, Stop/SubagentStop) the action is blocked. This is how a PreToolUse hook vetoes an rm -rf. (For PreToolUse specifically there's now a cleaner path too: emit permissionDecision — allow, deny, or ask — inside hookSpecificOutput on exit 0, which is more expressive than the blunt exit-2 veto and lets you attach a reason the model reads. I still reach for exit 2 when I just want a hard no.)

Any other exit code — non-blocking error. The session continues; the failure is surfaced in the transcript and logged, but nothing is blocked.

The JSON you emit on stdout (with exit 0) has a couple of shapes. There's a set of top-level universal fields — continue (set false to stop Claude entirely), stopReason, suppressOutput, systemMessage. And there's an event-specific envelope, hookSpecificOutput, which is where the good stuff lives. The single field I use most is additionalContext: a string that Claude Code injects into the model's context at the point the hook fired.

{
  "hookSpecificOutput": {
    "hookEventName": "SessionStart",
    "additionalContext": "Current branch: main\nUncommitted: auth.ts, config.py"
  }
}

That's the whole trick behind hook-injected memory. additionalContext on a SessionStart hook lands at the start of the conversation. The same field on UserPromptSubmit lands next to the prompt the user just submitted. On PreToolUse/PostToolUse it lands next to the tool result. The docs render it as a system reminder and advise writing it as plain factual statements rather than instructions — the model treats "the deployment target is production" better than "remember to be careful about production." There's a cap on how much you can push through it (on the order of ten thousand characters), which is less a limit than a hint: injection is not a place to dump files.

The one rule that fails silently. If you emit JSON on stdout with exit 0, stdout must contain only that JSON. A stray echo from your shell profile, a debug print, a warning from a Python import — any of it corrupts the JSON and your injection silently does nothing. More than one of my early hooks failed for exactly this reason and gave no error, because a malformed stdout on exit 0 just means "no decision," not "error." Nothing tells you. The session simply proceeds as if the hook weren't there.

In the interactive version of this post there's a small explorer where you pick an event, an exit code, and a stdout shape and see exactly what Claude Code does — including how the "stray echo" case turns an injection into a silent no-op. It's on swapnanilsaha.com.

A real pipeline: injecting working memory

Now the case study. My tool is a working-memory MCP server: the agent stores notes during a session and recalls them later. The problem from the intro was that recall is a tool the model has to choose to call, and it wouldn't, reliably. Hooks are how I took the choice away from the model and made recall happen deterministically.

When you run vectr init --hooks, the tool writes four hook groups into the project's .claude/settings.json. Every one of them calls back into the same CLI — vectr hook <event> — which owns the output contract so the settings file stays a thin, stable pointer. Here's the shape it writes (the install code is idempotent — re-running never duplicates entries and leaves any hooks you added yourself untouched):

{
  "hooks": {
    "SessionStart": [
      { "matcher": "startup|resume|clear|compact",
        "hooks": [{ "type": "command", "command": "vectr hook session-start" }] }
    ],
    "UserPromptSubmit": [
      { "hooks": [{ "type": "command", "command": "vectr hook user-prompt-submit" }] }
    ],
    "PreToolUse": [
      { "matcher": "Edit|Write",
        "hooks": [{ "type": "command", "command": "vectr hook pre-tool-use" }] }
    ],
    "PreCompact": [
      { "matcher": "manual|auto",
        "hooks": [{ "type": "command", "command": "vectr hook pre-compact" }] }
    ]
  }
}

Four events, four jobs. Walk through why each one is the event it is.

SessionStart — the boot set

Before the agent's first turn, vectr hook session-start fires. It resolves which of my running daemons serves this workspace, asks it for the boot set — the must-see notes, meaning standing directives plus high-priority task context — and emits them as additionalContext. This is the MEMORY.md-equivalent: the handful of things that should be true in the agent's head from turn one, present with zero model agency. The matcher startup|resume|clear|compact means it fires not just on a fresh start but also after a /compact and after a /clear — precisely the moments when the agent has just lost its context and most needs the boot set re-injected.

UserPromptSubmit — per-turn recall

This is the one that fixed the original problem. Every time the user submits a prompt, vectr hook user-prompt-submit reads the prompt text off stdin, runs a semantic recall against the note store keyed to that specific prompt, and injects the top matches next to the prompt before the model ever sees it. Ask about workspace locking and the locking notes are already there. The agent doesn't decide to recall — recall already happened, invisibly, on the way in.

The tuning here matters because this hook is on the hot path of every single turn. I cap it hard: at most 3 notes, with a relevance floor (a minimum similarity of 0.35) so an off-topic prompt injects nothing rather than dragging in vaguely-related noise. And it injects the terse one-line index form of each note, not the full body — enough for the model to know the note exists and decide whether to expand it, without spending a paragraph of tokens on every turn. An injection that fires every turn has to be miserly or it becomes the context bloat it was meant to prevent.

PreToolUse (Edit|Write) — the gotcha at the moment of the edit

This one I'm quietly proud of. When the agent is about to edit or write a file, vectr hook pre-tool-use pulls the file_path out of the tool input and recalls any gotcha recorded against that exact file — then injects it right there, at the instant of the edit. "This file's config is regenerated; edit schema.ts instead." "This function looks unrelated but changing it breaks the lock invariant." Static path-scoped rules can't do this, because the gotcha is something an earlier session learned and wrote down, and it surfaces exactly when it's actionable rather than sitting in a rules file the agent skimmed once.

PreCompact — save it before it's gone

/compact replaces the conversation with a summary and, in doing so, throws away exact detail. So right before it runs, vectr hook pre-compact snapshots the working-memory store — sealing the current notes as a named checkpoint. Notably this hook injects nothing into context; compaction is about to discard context anyway, so there's no point. Its whole job is the side effect of persisting state, and the boot set gets re-injected on the other side by the SessionStart compact matcher. The two hooks are a matched pair around the compaction event: one saves, the other restores.

Why hook-injected memory beats a recall tool

Let me make the central design argument sharp, because it's the reason the pipeline exists in this shape.

A recall tool and a recall hook retrieve the exact same notes from the exact same store. The only difference is who pulls the trigger. With a tool, the model decides — and "the model decides" means a probability, well short of 1, that it happens on any given turn, dropping further as the task gets absorbing. With a hook, the harness decides, and the harness is deterministic. It fires every time, on schedule, whether or not the model would have thought to.

For a capability whose entire value proposition is reliability across sessions, a probabilistic trigger is a contradiction in terms. Working memory you recall 60% of the time isn't 60% as good as working memory you recall always — it's worse than that, because the times it fails are unpredictable and the agent has no way to know it's operating on a stale or empty picture. Moving the trigger from the model into the harness is the difference between a feature that demos well and one that holds up.

The arithmetic makes it concrete. If a recall tool fires with probability p on each turn, the chance it fires on every turn of an N-turn session is p^N. At p = 0.60 over 20 turns that's about 0.004% — a clean session is essentially impossible. Even a very obedient p = 0.95 gives you only about 36%. A hook is p = 1, so p^N = 1, every session, forever.

The general principle. Anything that must happen every time belongs in a hook, not in an instruction. A guarantee cannot be prompt-engineered, because the thing you'd be prompting is the exact thing you don't control. Relocate the trigger, not the words.

There's a subtle second-order bug that falls out of doing this, and it's worth telling because it's the kind of thing you only find in real transcripts. Once SessionStart and UserPromptSubmit are auto-injecting notes, the model — which has also been told in CLAUDE.md to recall notes — will sometimes call the recall tool on top of the injection, paying for the same memory twice. I caught this in an eval transcript: the agent got its notes injected by the hook and then immediately called recall for the same thing. The fix is a one-line notice prepended to the injected context: "Your working-memory notes are auto-injected below — do not call recall to re-fetch them; call it only for something not shown here." It resolves the double-dip cleanly, but I'd never have known to write it without watching the failure happen.

The three things that will hurt you

Hooks are simple to write and easy to write dangerously. Three concerns dominate, and they're all about what happens when a hook misbehaves.

1. A hook must never break the session

This is the rule I hold most rigidly, and it shapes every line of my hook code. A hook runs on the critical path — UserPromptSubmit fires before every prompt the user sends. If that hook throws, hangs, or crashes, it degrades or breaks the user's session. So the hook code is paranoid by construction:

The recall function that feeds the injection catches every exception and returns an empty string on any failure. Daemon down, slow, error, malformed response — doesn't matter, it yields nothing and the session proceeds.
The top-level hook handler wraps its entire body in a try/except and always exits 0. There is no code path where my hook returns a non-zero exit and accidentally blocks a prompt or a tool call.
If there's no memory to inject — a brand-new workspace with zero notes — the hook emits nothing at all, not an empty JSON envelope. A fresh project should feel exactly like no hook is installed.

The design stance is that the memory injection is a bonus, never a dependency. The session must work identically whether the daemon is up, down, or on fire. If your hook can make the agent worse when it fails, you've built a liability, not a feature.

2. Latency is a tax on every turn

UserPromptSubmit sits between the user hitting enter and the model starting to think. Whatever your hook spends there, the user waits. Claude Code caps this event's hook timeout lower than others for exactly this reason, but a timeout is a backstop, not a budget — you want to be nowhere near it. My recall is designed to return in well under 50 milliseconds, and the hook does the absolute minimum: read stdin, one local HTTP call to an already-running daemon, print, exit. No model loading, no indexing, no network beyond localhost. If your per-turn hook does anything that can take a second, move it off the hot path — make it async, or attach it to a less frequent event.

3. Hooks run arbitrary shell — treat them as such

The official docs are blunt about this and they're right: hooks execute arbitrary shell commands with your full user permissions, automatically. They can read your files and your environment variables. A malicious or careless hook in a shared .claude/settings.json is a genuine attack surface — someone commits a hook, you pull the repo, and now their command runs on your machine the next time you start a session.

Practical defenses: review hook configs before committing to shared repos, exactly as you'd review a Makefile or a git hook; keep personal hooks in the gitignored settings.local.json so they can't leak; and know that enterprise setups can lock this down with allowManagedHooksOnly. In my own design I lean on a smaller mitigation — the settings file never contains logic, only vectr hook <event>, a call into a versioned, inspectable CLI. There's no shell one-liner in the JSON to audit; the behavior lives in code you can read. And the CLI only ever talks to a localhost daemon, so a hook firing in the wrong directory can't reach across to another workspace's memory. That last point is deliberate: the resolver walks up from the current directory to find the daemon that serves this workspace and refuses to fall back to a default — because a default port could belong to an unrelated project and leak its notes into your session.

The honest limitations

A few things I've hit that the enthusiastic tutorials leave out.

Injected context is still context. Every note a hook injects costs tokens, every turn, forever. The UserPromptSubmit hook is genuinely helpful because it's disciplined — 3 notes, relevance floor, terse index form. An undisciplined version that injected ten full notes per turn would reintroduce the exact context bloat the memory system exists to fight. Hook injection is a budget you're spending; spend it like one.

Debugging is opaque by design. Because a malformed stdout on exit 0 means "no decision" rather than "error," a broken hook fails silently. The session just proceeds as if the hook weren't there. When an injection isn't landing, my first move is always to run the exact command by hand, pipe a sample event JSON into its stdin, and stare at stdout for the one stray character breaking the JSON. There's no substitute; the harness won't tell you.

It's Claude Code-shaped. This whole mechanism is specific to one harness. My pipeline's determinism comes from Claude Code's hook system, and other agent environments have different injection points, or none. If you want the same deterministic-injection behavior elsewhere, you're re-implementing against a different (or absent) surface, and in the worst case you fall back to the very thing hooks let you escape — hoping the model calls the tool. That portability gap is real and I don't have a clean answer to it yet.

Compaction timing isn't fully in your hands. PreCompact fires before compaction, which is great, but auto-compaction triggers on the harness's schedule, near the context limit — not necessarily at a clean task boundary. My snapshot is a safety net, not a substitute for the agent proactively writing important findings to memory as it goes. The hook catches what the agent forgot to save; it works best when there's little to catch.

What hooks are really for

Strip away the specifics and hooks are one idea: a place to put behavior that must not depend on the model's cooperation. Injection, enforcement, persistence, cleanup — anything where "usually" isn't good enough and you need "always." The model is brilliant at the open-ended, judgment-heavy work in the middle of a turn. It is not the thing you want deciding whether the guardrail runs.

For my working-memory tool, hooks are what turned a good idea that demoed well into something that actually holds across sessions. The notes were always there. What was missing was a guarantee that the agent would look — and that guarantee cannot come from the agent. It comes from four small shell commands wired into the right four events, each one paranoid about never breaking the session, each one doing exactly one deterministic job. That's the whole art of it: not clever hooks, but reliable ones.

Companion posts

This post is part of a series on the working-memory tool the pipeline is built on:

Sources

Claude Code — Hooks reference (event list, settings.json structure, matchers, exit-code protocol, hookSpecificOutput/additionalContext, security warning, allowManagedHooksOnly). Accessed 2026-07-07.
Claude Code — Automate actions with hooks (worked configuration examples).
Claude Code — Context window, compaction, and /compact. Accessed 2026-07-07.

Building Vectr, Part 2: What /compact Destroys and How to Survive It

Swapnanil Saha — Tue, 16 Jun 2026 13:28:51 +0000

Session three of a bug hunt in CPython's garbage collector. Two sessions in, I had what felt like a solid map: the exact call chain from PyObject_GC_Del through the generational collector, the non-obvious invariant around finalizer ordering, the three files where the relevant logic lived. Then /compact fired.

The summary said something like: "we were investigating CPython's garbage collector, specifically the interaction between finalizers and the generational GC." Accurate. Useless. The exact function signatures were gone. The specific line numbers were gone. The invariant that took two sessions to understand — compressed to one sentence that had lost all the nuance. The next 20 minutes: re-reading files to rebuild what I already knew.

This post is about what I learned from that, and from the working memory system I built to prevent it. Part 1 covered the indexing layer — how Vectr finds things in a codebase semantically. This part covers what happens after you find something: how to keep the knowledge alive across session boundaries, why my initial design was wrong in a fundamental way, and what actually works.

Part 1: The Problem With /compact

What /compact Actually Destroys

Most people treat /compact as "clear the context to keep going." That framing is roughly correct but understates the damage. The issue isn't just that context gets shorter — it's that the compression is lossy in exactly the cases where being wrong is most expensive.

/compact works by asking the AI to summarize the current conversation, then replacing the full history with that summary. Token count drops from (say) 180,000 to 12,000. Here's what the summary doesn't preserve:

Exact function signatures. A summary might say "the function takes a path and a flag." The conversation had def process_workspace_changes(path: Path, db: Database, *, force: bool = False) -> list[ChangeResult]. The difference between those two descriptions is the difference between a valid call site and a runtime error.

Specific line numbers. "The resolver module" and /src/workspace/resolver.rs:214 are not the same precision. You can reconstruct the file path, but it costs you a tool call.

Non-obvious behavioral invariants. If you spent three turns establishing that acquire_lock() must be called before touching workspace metadata because there's a race condition with the filesystem watcher, that three-turn understanding might survive as "be careful with locking." The exact invariant — the one that matters when you're writing the code — is gone.

The reasoning chain. Sometimes the value of an exploration session isn't the final answer but the chain of observations that produced it. Summaries discard chains. They keep endpoints.

Key insight: Summaries are fine for preserving topics and general direction. They fail specifically at exact signatures, line numbers, and subtle behavioral invariants — which is also where being wrong is most expensive. A summary of "be careful with locking" covers the topic. It doesn't tell you which function must be called first, or why, or what breaks if you get it wrong.

In the CPython scenario, re-establishing the finalizer ordering invariant from scratch means re-reading several files and re-following a non-obvious call chain — roughly 15–20 minutes of work that was already done. A note stored at the end of session two takes about a minute to write and ten milliseconds to retrieve.

Why You Can't Tell the AI to Just Forget Things

When I started building Vectr's memory layer, I had a clean model: the AI finds something useful, stores it with vectr_remember, then drops the file from its context window. The note is 50 tokens. The file was 800 tokens. Net gain: 750 tokens freed for new content. I called this "context offload."

I built it this way. I wrote documentation describing it this way. I designed vectr_evict_hint entirely around it.

It doesn't work.

The KV cache is append-only. Think of the transformer's memory as a lookup table it builds as it reads each token. For each token it processes, it computes a key-value representation that gets stored at each attention layer. Every subsequent token attends back to every previous token through these cached representations — that's how earlier context influences later output.

Once a token's representation is computed and cached, it stays until the context is cleared. There is no mechanism to evict specific tokens by instruction. "You can drop chunk X from your context window" is itself processed as tokens — added to the cache, not used to remove other entries from it.

A subtlety worth naming: the KV cache is maintained server-side by the inference provider. What you see as "context window usage" is a count of tokens in the current conversation, not a direct readout of GPU memory. The principle holds regardless: every token in the conversation occupies a slot in the cache, and you cannot remove individual tokens from a running session without ending or compressing the whole thing.

The KV cache memory cost formula:

KV cache size = 2 × L × n_heads × d_head × T × bytes_per_float

For a representative mid-size model: L=32 layers, n_heads=32, d_head=128, T=50,000 tokens at fp16 (2 bytes):

2 × 32 × 32 × 128 × 50,000 × 2 = 13.1 GB

The cache grows linearly with sequence length T. No selective removal. The operations that genuinely reduce context are: end the session (total loss), use /compact (precision loss), or rely on provider-side prefix caching — which stores stable prefix representations like system prompts to avoid recomputing them, but doesn't remove anything from your active context budget.

I measured context window usage before and after sequences of vectr_remember + vectr_evict_hint calls: essentially unchanged. The hint was adding tokens to the cache while accomplishing nothing at the context management level. In some cases it made things marginally worse.

Warning: Any tool or documentation claiming "store to external memory to free context budget" is describing something the system cannot deliver. Tokens in a live context window cannot be selectively evicted. Working memory tools are genuinely valuable — but not for freeing active context. Building around that claim confuses your benchmarks and misleads anyone using the tool.

Part 2: What Working Memory Actually Does

Three Tiers of Value

Once I dropped the context-offload framing, the actual value of vectr_remember became clear. It operates on three time horizons:

Tier 1 — In-session re-read avoidance. Within a single session, before any /compact: recalling a stored note costs ~50 tokens instead of re-reading the original file at ~600 tokens. Real savings, but the file is still sitting in your context window anyway. Genuinely useful, but not the reason to build this.

Tier 2 — /compact survival. When /compact compresses the conversation, notes stored on disk (SQLite + ChromaDB) are untouched. Exact signatures and behavioral invariants survive verbatim. The session resumes from actual precision. This is where the system earns its cost.

Tier 3 — Cross-session persistence. Between separate sessions — the editor closed and reopened — the AI starts with nothing. Notes survive. A new session calling vectr_status() + vectr_recall() recovers findings from sessions ago without re-reading a single file. Each session builds on the ones before it.

Analogy — The surgeon's notes: A surgeon takes detailed notes before starting a complex procedure. Halfway through, an emergency calls them away for two hours. When they return: (a) their notes are on the desk — exact measurements, named vessels, where they left off; or (b) a colleague wrote a summary: "patient is partially through a vascular procedure, some complications noted." Option (b) is dangerous. Option (a) lets you continue precisely. vectr_remember is option (a). /compact without notes is option (b).

Tier 3 compounds in a way that's easy to underestimate. The first session on a complex codebase pays the discovery cost. The second benefits from the first session's notes. By the tenth session, a well-maintained note store is a persistent model of the codebase that makes every session faster.

What to Store and How

Don't store file pointers. "See resolver.rs:214 for the lock implementation" is a bad note. File paths change during refactoring. Line numbers drift with every edit. A pointer hasn't captured what you learned — it's a reference. When you recall it, you still have to read the file.

Store the finding itself:

WorkspaceLock: defined at resolver.rs:214 (as of 2026-06-08)
- acquire(): blocks if .vectr_lock exists; writes current PID + timestamp
- release(): validates PID match before deleting lock file
  (returns Err if mismatch — this is intentional, not a bug)
- CRITICAL: acquire() must be called BEFORE touching workspace
  metadata. The filesystem watcher reads metadata; touching it
  without holding the lock fires an invalid re-index.
  This caused the race condition in issue #1247.

Key callsites: workspace.rs:89 (init), daemon.rs:203 (shutdown)

This note is ~120 tokens. Reading the relevant files to reconstruct this knowledge would cost 600+ tokens plus two turns. The note captures the actual insight — the non-obvious invariant about lock order — not just a pointer.

Priority and tags are not cosmetic. priority affects recall ordering: high-priority notes rank higher when multiple notes match a query with similar scores. tags enable filtered recall — vectr_recall(query="locking", tags=["concurrency"]) returns only notes tagged with "concurrency" that semantically match the query. In a large note store accumulated over months, filtering by subsystem makes recall precise.

Part 3: The Bugs That Shaped the Design

The B9 Bug: When Recall Doesn't Recall

For several early benchmark runs, vectr_recall was firing in implementation sessions but returning nothing useful — 0 relevant results across 5 separate sessions on CPython tasks, even though the research session had stored detailed notes about exactly the functions being modified.

Root cause: recall was using SQL LIKE queries, not semantic search.

# The broken implementation (pre-B9)
def recall(query: str) -> list[Note]:
    return db.execute(
        "SELECT * FROM notes WHERE content LIKE ? LIMIT 20",
        (f"%{query}%",)
    ).fetchall()

SQL LIKE is substring matching. vectr_recall("garbage collector finalizer ordering") would only return notes containing that exact string. A note about PyObject_GC_Del describing finalizer behavior — stored with different wording in a different session — wouldn't match.

The fix: use the ChromaDB vector store for recall. Notes are embedded when stored, retrieved by semantic similarity when recalled.

# The correct implementation (post-B9)
def recall(query: str, tags: list[str] | None = None) -> list[Note]:
    results = chroma_collection.query(
        query_texts=[query],
        n_results=10,
        where={"tags": {"$in": tags}} if tags else None,
    )
    return [Note.from_chroma(r) for r in results]

Impact was immediate: vectr_recall fired with relevant results in 4 of 6 implementation sessions in the CPython re-run, compared to 0 of 6 before. This bug sat undetected because the initial benchmark design didn't make empty recalls visible. Per-tool logging — "vectr_recall called 5 times, 5 empty responses" — made it obvious.

Warning: SQL LIKE requires the query string to be a literal substring of the stored content. For anything more than exact-match lookup, it's not just suboptimal — it's functionally broken for most real queries.

vectr_evict_hint: What It Actually Does After the Reframe

After fixing the context-offload misconception, I kept vectr_evict_hint but reframed it completely. What it actually does: it tracks the cumulative token cost of all code chunks Vectr has retrieved in the current session. When this cost crosses a threshold (40K tokens or 20 tool calls — whichever fires first), it appends a hint:

[vectr_evict_hint] You've retrieved ~42,000 tokens of indexed chunks
this session. The following chunks are fully indexed and re-retrievable
in <50ms — no need to re-read these files later:

  - resolver.rs:214  WorkspaceLock::acquire  (retrieved 8 turns ago)
  - resolver.rs:267  WorkspaceLock::release  (retrieved 8 turns ago)
  - workspace.rs:89  init call site          (retrieved 5 turns ago)

Consider calling vectr_remember now if you have key findings you
haven't stored yet.

The word is "re-retrievable," not "droppable." The hint doesn't claim to free tokens. It tells the AI: these files are in the index, you can get them back in under 50ms if you need them — don't re-read out of caution when you already have what you need or could re-search instantly. It's a behavioral nudge, not a memory management operation.

The threshold values come from MemGPT (arXiv:2310.08560), which found models begin exhibiting "lost in the middle" degradation at roughly 70% context fill. Using a disjunction (first threshold reached triggers the hint) keeps it from firing too late on sessions that accumulate few large files but many small searches.

Lost in the middle: LLM performance on retrieval tasks follows a U-shaped curve over context position — accuracy highest at the beginning and end, degrading for content in the middle. The evict_hint threshold is set to fire before relevant information drifts into that degraded zone.

Part 4: The Mechanics of Actually Using It

The Save-Moment Problem

Knowing notes are valuable doesn't make the AI store them. In early sessions, vectr_remember call rates were low — not because the AI couldn't see the tool, but because there was no clear trigger for "now is the moment to save this."

Saving notes is a habit humans develop from experiencing loss. An AI editor in session 1 has never lost anything to /compact here — it's optimizing for the task in front of it, not a compression event that might happen three hours from now.

The solution: making the save-moment explicit and concrete in the CLAUDE.md template that Vectr writes into a workspace.

**The moment you find a key definition, pattern, or non-obvious detail:**
call vectr_remember(content, tags=[...], priority="high"|"medium"|"low")
— store the actual code block or finding, not a file pointer.

Treat every vectr_search or vectr_locate call as a **pair**: search,
then immediately save the key finding before your next retrieval.

If /compact runs later, the conversation summary loses exact signatures
and line numbers — your note does not.

"Pair every search with a save" turned out to be the most effective framing. Not "save when it feels important" (too vague), but "pair every retrieval with a note" (concrete, immediate trigger). Sessions that stored the most notes also had the lowest re-discovery costs in subsequent tasks.

When not to search: the SR-RAG finding. The pair pattern addresses when to save. There's a complementary question that ended up in the same CLAUDE.md template: when to search at all. Before calling vectr_search on a well-known API or framework, the AI should first write out what it already knows and only search if genuine gaps remain.

This comes from SR-RAG (arXiv:2504.01018). The finding: models often retrieve information already baked in from training, adding token cost without improving answer quality. Writing out what you already know before searching reduces unnecessary calls by 26–40% on familiar codebases. On an unfamiliar codebase, the AI's training knowledge rarely applies — every search turns up something new. On well-known frameworks, training knowledge is often more accurate than indexed documentation. The verbalization step surfaces which situation you're actually in.

Snapshots: Checkpointing an Investigation

Beyond individual notes, there's a use case for checkpointing entire session states. vectr_snapshot("lock-subsystem-mapped") seals the current note set under a named label with a timestamp. vectr_snapshot_list() at session start shows all checkpoints.

Typical multi-session workflow:

Exploration sessions: explore, call vectr_remember on each key finding. Pair every search with a save.
Exploration complete: vectr_snapshot("exploration-complete"). Seals the note state for this phase.
Implementation sessions: vectr_status() → vectr_recall(query) → build on the snapshot.
Implementation done: vectr_snapshot("implementation-done"). Two named checkpoints marking the arc.
Revisiting months later: vectr_snapshot_list() shows the investigation history. The snapshot timestamp tells you which notes were established before a given change.

When Notes Are Wrong: vectr_forget

Notes can be wrong. A note about function behavior written before a refactor may describe the old behavior. Stale notes are worse than no notes — false confidence in outdated information.

vectr_forget(note_id) deletes it. Every vectr_recall response includes note IDs alongside the content so you can act on them inline. The workflow: recall → verify against current code → forget the stale note → store the updated one.

Vectr also appends a [STALE] marker automatically when a file path extracted from a note's content no longer exists in the workspace. The extraction is a regex scan for path-like strings — when those paths disappear from the file tree, the note gets flagged. It only catches path-level staleness, not behavioral changes in files that kept their names.

Warning: The [STALE] marker fires when a referenced file path disappears. It does NOT fire when file content changes. A note about function behavior after a refactor that renamed the file gets flagged; a note about function behavior after a refactor that changed the logic without renaming gets no warning. Always verify behavioral notes against current code before acting on them for implementation work.

The Design Principle I'd Rephrase

Looking back at the original Vectr documentation for working memory, almost every sentence led with the wrong framing. "Store to vectr, then drop from context." "Offload findings to free context budget." "Context offload layer." Every one of these is technically false, and I shipped all of them.

The correct version is shorter: store findings now so you can recall them precisely later. Through /compact. Through a new session. Through however many turns separate the discovery from the moment you need to use it. The value is in the later. The storing is cheap. The recalling is where you get the hours back.

If I were writing the documentation from scratch I'd lead with the /compact scenario — with the specific moment when a detailed understanding of a complex system compresses into a three-sentence summary that can't be acted on. That's the moment where a stored note is worth exactly what it cost to write it.

What's Next

The part I haven't answered yet: does any of this actually save time? Not in the abstract — in real benchmarks, on real codebases, compared against an AI editor with no indexing and no memory. The number I care about is not total session cost (which includes upfront research overhead that inflates the naive comparison) but re-discovery cost per task across repeated sessions on the same codebase.

Part 3 covers that measurement — including why the total sprint cost comparison is almost exactly the wrong metric to report, and what the data from CPython, Django, and Apache Camel actually showed once I separated research overhead from implementation savings.

If you want to try Vectr now, the tool page has setup instructions. The full working memory layer — vectr_remember, vectr_recall, vectr_snapshot, vectr_forget — is in the current release alongside the semantic search tools from Part 1.

References

Packer et al., MemGPT: Towards LLMs as Operating Systems, arXiv:2310.08560, 2023
Liu et al., Lost in the Middle: How Language Models Use Long Contexts, arXiv:2307.03172, 2023
Self-Routing RAG: Binding Selective Retrieval with Knowledge Verbalization, arXiv:2504.01018, 2025
Vaswani et al., Attention Is All You Need, NeurIPS 2017
A Survey on LLM Acceleration Based on KV Cache Management, arXiv:2412.19442, 2024

Building Vectr, Part 1: Why grep Fails When You Don't Know the Keywords

Swapnanil Saha — Tue, 09 Jun 2026 14:10:06 +0000

This is Part 1 of the Building Vectr series (1 of 3).

You get dropped into an unfamiliar codebase. Not a toy project — real production code, 8,000 files, three years of accumulated complexity and clever abstractions. Your job is to fix a bug in the request validation pipeline. What does an AI code editor do next?

This post is about a problem I kept running into, a tax I kept paying, and the indexing system I built to eliminate it. It covers the technical decisions behind Vectr's search layer: why naive chunking produces bad embeddings, how tree-sitter solves the code-parsing problem, what BM25 does that vector search can't, and why you need a symbol graph for questions that text search cannot answer at all.

Part 1 — The Problem

The Re-discovery Tax

If you're a human engineer navigating an unfamiliar codebase, here's what you probably do: you ask someone who knows it, or you grep for the error message, or you open the entry point and follow imports until you find the thing. Your brain does semantic compression the whole way — building a model of the system, discarding noise, following intuitions about where complexity tends to live. By the time you've read 20 files, you have a rough map that persists across days and sessions.

An AI code editor has the same tools — read files, run shell commands, grep — but completely different economics. Every Read call costs tokens. Every Bash call for grep costs a turn. Unlike a human who can skim-read at 1,000 words per minute and discard irrelevant content almost for free, an AI editor pays full price for every character it reads: it sits in the context window whether or not it was useful. Read the wrong 500-line file and you've burned context that could have held the answer.

The result, on unfamiliar codebases, is what I started calling the re-discovery tax: a cluster of navigation calls at the start of every session, before any actual implementation begins, spent on figuring out where things are. And because AI editors have no persistent memory between sessions, they pay this tax again and again — every session, on the same codebase.

In benchmarks I ran against real open-source codebases (more detail in Part 3), the re-discovery tax on CPython internals ranged from 6 to 23 tool calls per task before the first file write. Some sessions spent more turns navigating than implementing.

Key observation: The re-discovery tax is paid every session, not once. A human engineer's mental map of a codebase accumulates and compounds. An AI editor's map is fully rebuilt from scratch at the start of each session. The economic gap widens as the codebase grows.

Why grep Fails at the Boundary of Your Knowledge

Before explaining what I built, I want to be precise about where grep breaks down — because "just use grep" is the natural reaction, and it's not obviously wrong until you try to use it systematically on unfamiliar code.

grep is a brilliant tool for confirming hypotheses you already have. If you know what you're looking for, it's nearly perfect. The problem is the case that isn't really an edge case: you don't know what you're looking for.

Say you're trying to understand how a Django application validates incoming JSON payloads before they hit the ORM layer. You might grep for validate. You'll get 200 results across 40 files — field validators, form validators, configuration validators, test fixtures. None of them are obviously the thing you want. You grep for json.loads. You get 30 results. You grep for request.data. That gets you closer, maybe. But you spent four greps and 15 minutes before you found the right file.

The deeper problem: grep requires you to already have a mental model of the codebase's naming conventions. An AI editor running on an unfamiliar codebase doesn't know whether payload validation is called validate_payload, check_request, parse_input, or _pre_process.

Analogy: Think of keyword search as asking for directions by street name in a city you've never visited. "Where is Maple Street?" gets a precise answer. But "where is the street with the good coffee shop near the park?" — keyword search has nothing to offer. You need a different kind of index: one that understands what places are for, not just what they're called.

Semantic search inverts this. It maps your query and every code chunk into the same high-dimensional vector space, then finds the chunks closest to your query by meaning — regardless of whether they share any words. "JWT validation logic" finds verify_token even if neither of those words appears in the function body.

Part 2 — Building the Index

The Chunking Problem: Why Line Windows Break on Code

Prose text has a natural unit of meaning: the paragraph. You can split a Wikipedia article into 200-word chunks, embed each one, and get a reasonable search system. Code doesn't work this way.

The standard naive approach for code indexing is the same line-window strategy borrowed from document search: take a sliding window of N lines with M lines of overlap, create a chunk, embed it, move the window. A common default might be 150-line windows with 50 lines of overlap. Simple, language-agnostic, works on any file format.

The problem is what happens at the window boundaries. Consider this function:

def process_workspace_changes(
    path: Path, db: Database, *, force: bool = False
) -> list[ChangeResult]:
    """Process all pending changes in a workspace, optionally forcing re-indexing."""
    pending = db.get_pending_changes(path)
    if not pending and not force:
        return []

    results = []
    for change in pending:
        if change.kind == ChangeKind.DELETED:
            db.remove_chunks_for_file(change.file)
            results.append(ChangeResult(file=change.file, status="removed"))
        elif change.kind in (ChangeKind.MODIFIED, ChangeKind.CREATED):
            chunks = chunk_file(change.file, db.language_for(change.file))
            db.upsert_chunks(chunks)
            results.append(ChangeResult(
                file=change.file, status="indexed", chunk_count=len(chunks)
            ))

    db.mark_changes_processed(path)
    return results

If a 150-line window happens to cut through this function, neither resulting chunk is independently meaningful. The chunk with just the body is missing the parameter names and return type. The chunk with just the signature has no implementation context. The embedding of a half-function is significantly worse than the embedding of the complete thing.

The fix: split at semantic boundaries. Functions should be complete units. Classes should contain their methods, or each method should be its own chunk with the class header prepended for context.

Why completeness matters: An embedding model compresses everything in its context into a single fixed-size vector. A complete function gives the model everything it needs to capture the function's purpose, parameters, return behavior, and side effects in that vector. A half-function forces the model to compress an ambiguous fragment — the resulting vector is a blurred average of possible interpretations.

Parsing Code with tree-sitter

tree-sitter is a parser library that produces concrete syntax trees for source code — every construct in the language has a named node with exact byte boundaries in the source. Unlike a regex-based approach, tree-sitter actually parses the grammar and handles edge cases correctly: nested functions, decorators on multiple lines, multiline function signatures, arrow functions in JavaScript, generic bounds in Rust.

For Python, the tree-sitter query:

(function_definition
  name: (identifier) @name
  parameters: (parameters) @params
  body: (block) @body) @function

(class_definition
  name: (identifier) @name
  superclasses: (argument_list)? @bases
  body: (block) @body) @class

This matches any function or class definition anywhere in the file and captures the name, parameters, and body as named nodes with precise byte-range positions. You can then slice the original source file at those byte positions to extract complete, syntactically valid chunks.

For classes, Vectr attaches the full class signature — including the base class list captured by @bases — as a header to each method chunk. So the chunk for WorkspaceLock.acquire() includes its inheritance context. A method of AuthenticatedView(LoginRequiredMixin, View) has a meaningfully different semantic context than a method of a plain View.

A subtlety: very large functions. AST-aware chunking breaks down for functions that are genuinely enormous — 500+ lines. Vectr handles this by further splitting large functions at their major control-flow boundaries (default threshold: 200 lines). The resulting sub-chunks each include the function signature as a header to preserve context. Their embedding quality is better than one giant embedding, though still lower than a naturally small function.

Code-Specific Embeddings Running Locally

Not all embedding models are equally good at code. Models trained primarily on prose text have learned representations of natural language semantics. Code has different regularities: symbol names, type signatures, control flow patterns, API call chains. Code-aware models routinely outperform general-purpose models by 10–20% on tasks like "find the function that handles X."

Vectr uses Snowflake/snowflake-arctic-embed-m-v1.5, a 110-million-parameter model that produces 768-dimensional embedding vectors and runs in under 100ms per batch on a modern laptop CPU.

Why local inference instead of an API? Two practical constraints:

Cost: a tool that fires 20–50 search calls per session would accumulate non-trivial API costs quickly. Local inference is free at query time after the one-time model download.
Data privacy: many codebases cannot be sent to third-party APIs. Internal tools, proprietary algorithms, customer data models — many organizations have policies or contractual obligations that prohibit sending source code to external services.

The tradeoff: the model weighs roughly 440MB and needs to be downloaded on first run. This is a real friction point.

One critical detail: queries and chunks are embedded with different input prefixes. Queries use Represent this query for searching relevant code:, chunks use Represent this code snippet:. arctic-embed-m is a single encoder, but it was trained with different prefixes for query-side and document-side inputs. Using the wrong prefix reduces the cosine similarity between semantically related query-chunk pairs — the vectors for "user authentication" and verify_token end up further apart in embedding space than they should be. Getting this wrong costs 5–15% in retrieval quality.

Part 3 — The Search Layer

Hybrid Search: Why BM25 and Vector Search Need Each Other

Vector search handles concept queries well. But if you search for _handle_workspace_lock_conflict — an exact function name — a vector search might not rank it first. The embedding is just one point in a crowded neighborhood of similar-looking function names. BM25, on the other hand, will find it immediately: exact string matches get the highest possible score.

The inverse is also true: BM25 cannot find "retry logic with exponential backoff" if the function is called _schedule_attempt_with_delay and its docstring says nothing about backoff. Zero keyword overlap means zero BM25 score. Vector search finds it because the semantic cluster it belongs to is close to the query in embedding space.

The right system uses both. Every query in Vectr runs both a vector search and a BM25 search in parallel, then combines the two ranked lists using a weighted formula.

BM25 scoring formula:

score(D, Q) = Σᵢ IDF(qᵢ) · [ tf(qᵢ, D) · (k₁ + 1) ] / [ tf(qᵢ, D) + k₁ · (1 − b + b · |D| / avgdl) ]

IDF(qᵢ) = log( (N − nᵢ + 0.5) / (nᵢ + 0.5) )

Where:

tf(qᵢ, D) — term frequency of qᵢ in document D
N — total documents; nᵢ — documents containing qᵢ
|D| — document length in tokens; avgdl — average document length
k₁ = 1.5 (term-frequency saturation), b = 0.75 (length normalization)

This is the Robertson–Sparck Jones variant. Some implementations add +1 inside the IDF log to prevent negative values for very common terms.

The weight assigned to each approach depends on codebase familiarity:

Situation	BM25 weight	Vector weight
Large unfamiliar codebase	0.2	0.8
Small familiar codebase	0.7	0.3
Explicit symbol name in query	0.8	0.2
Natural language concept query	0.2	0.8

These weights are the actual values used in Vectr's implementation, tuned against the benchmark dataset.

The benchmark on Apache Camel (58,000+ Java files) showed a 73% reduction in Read+Bash navigation calls compared to the baseline AI editor with no index.

The Symbol Graph: What Text Search Cannot Answer

Semantic search and BM25 handle "find me the code for this concept" well. But there's a different navigation pattern that neither handles: "find me everything that calls this function."

Vectr builds a symbol graph during indexing. For each file, tree-sitter extracts:

Definitions — every function, class, method, and module-level constant with name and line number
Call edges — every call site, mapping callee name to the calling function's context
Import edges — every import statement, mapping the imported symbol to its likely source module
HTTP routes — Flask/FastAPI @router.get(), Express app.post(), Spring @GetMapping — extracted as named symbols

The resulting graph enables exact lookups. vectr_locate("WorkspaceLock") returns a file path and line number in under 10ms — no embedding, no ranking, pure symbol table lookup. vectr_trace("acquire_lock") returns all callers and all callees in one round-trip. These are not search results — they are graph traversals, and they produce exact answers rather than relevance rankings.

Text search vs. graph traversal: These are not competing approaches — they answer different questions. "Find code that does X" is a search problem. "Find who calls Y" or "find where Z is defined" is a graph traversal problem. Relying only on text search for definition lookups is like looking up a phone number by describing the person rather than looking them up by name.

Six Fallback Strategies in vectr_locate

vectr_locate runs six fallback strategies in sequence, stopping at the first match:

Exact match — direct lookup in the symbol table. Sub-millisecond. Highest confidence.
Suffix match — Lock matches WorkspaceLock, AcquireLock, LockManager.
Same-module priority — if a caller file is provided, search definitions within the same module first.
Unique name — if there is exactly one symbol across the entire codebase whose name contains your query string, return it.
Import chain follow — follow import statements from a given file to find where the name likely comes from.
Fuzzy (Levenshtein ≤ 2) — edit distance ≤ 2 across all symbol names. Catches typos. Lowest confidence.

Each strategy produces a LocateResult with a resolution_strategy field. An exact match means you can act on the result immediately. A fuzzy match with edit distance 2 means you should verify before relying on it. A silent wrong navigation is worse than no navigation at all.

Part 4 — The Runtime Layer

mtime Cache and Incremental Re-indexing

The first time you run vectr start on a large codebase, indexing takes time. CPython's 4,000+ files: about 8 minutes. Django's ~1,800 Python files: about 2 minutes. Apache Camel's 58,000+ Java files: closer to 45 minutes.

During initial indexing, Vectr writes a file at ~/.cache/vectr/{hash}/index_cache.json that stores the modification timestamp of every indexed file. The {hash} is a short SHA-256 hash of the absolute workspace root path. On subsequent runs, only files whose mtime has changed are re-indexed. On a typical active session where you've modified 5–10 files, subsequent re-indexing takes under 5 seconds.

Handling deletions: Vectr also stores the complete set of indexed file paths. At startup, it diffs this set against the current file tree and removes all chunks belonging to deleted files before re-indexing modified ones. Process deletions first, then updates, then new files — this prevents a renamed file from leaving orphaned chunks in the index.

The watchdog listener: During an active session, Vectr runs a watchdog filesystem listener on the workspace root. When a file is saved, the listener queues it for re-indexing in the background. Events are debounced at 300ms — only the last write in a burst counts. Without debouncing, a single save in a project using aggressive auto-formatting would trigger 3–5 redundant re-index operations.

.vectrignore: Keeping the Index Clean

Vectr reads a .vectrignore file from the workspace root using glob patterns. The syntax follows .gitignore conventions — trailing slash for directories, * for single-level wildcard, ** for recursive match (via Python's pathlib.Path.match()) — but Vectr does not implement the full gitignore specification: the ! negation prefix is not supported.

vendor/
node_modules/
dist/
*.pb.go        # generated protobuf Go files
*.min.js       # minified JavaScript
__pycache__/
.venv/
coverage/
*.snap         # Jest snapshots
migrations/    # Django database migrations

A codebase with node_modules/ will typically contain 5–20x more code from installed packages than from the project itself. Excluding vendor directories before the initial index run is the single most impactful configuration change most users can make.

What Actually Happens When You Call vectr_search

1. Query string is embedded using arctic-embed-m with query prefix
   → 768-dimensional float vector, ~15ms on CPU

2. Vector similarity search against ChromaDB store
   → Top-20 chunks by cosine similarity, with scores

3. Same query runs through BM25 index (rank-bm25, in-memory)
   → Top-20 chunks by BM25 score, with scores

4. Two ranked lists are merged
   → Weight BM25/vector based on codebase characterization
   → Normalized scores combined; top-N results selected (default N=5)

5. Symbol names in the query are detected (camelCase, snake_case, PascalCase)
   → If found: also run vectr_locate as a side channel
   → Merge symbol lookup results into final output if relevant

6. Final top-N chunks returned with:
   file path, start line, end line, matched text, search method

Result for vectr_search("workspace lock acquisition and release"):

[1] resolver.rs:214 — WorkspaceLock::acquire()
    Acquires the workspace-scoped lock. Blocks if another process holds it.

[2] resolver.rs:267 — WorkspaceLock::release()
    Releases the workspace-scoped lock. Validates that the current process
    holds the lock before releasing (returns Err if not held).

[3] workspace.py:89 — _acquire_workspace_lock(path)
    Context manager: acquires, yields, releases on exit.

Instead of reading 15 files to find these three functions, the AI editor reads one search result.

Part 5 — Design Decisions I'd Make Differently

The Python 3.14 requirement. The codebase uses match/case pattern matching extensively and some asyncio patterns that behave differently in earlier versions. In retrospect, 3.11 would probably work with a few hours of refactoring. The 3.14 requirement has been the single biggest adoption friction.

ChromaDB as the vector store. A vector store handles embedding persistence and similarity search. ChromaDB works, but the full HNSW index with persistence, the Python client layer, and the inter-process communication overhead add about 200ms specifically to ChromaDB's startup contribution — not total Vectr startup (~280ms including mtime diffing and watchdog initialization). For v2, I'd consider a lighter in-process option.

The BM25 implementation. The rank-bm25 library is pure Python and fast enough for codebases under 50,000 chunks. Beyond that, it starts to show latency. The right long-term solution is integrating BM25 scoring directly into the vector store query pipeline. For current use cases (most codebases are under 20K chunks), it's fine.

Conclusion

The indexing layer is the foundation, not the product. What it enables is an AI code editor that can navigate a large unfamiliar codebase as efficiently as a human engineer who has worked in it for months — finding the right functions in one or two calls instead of fifteen.

But the index tells you where things are. It doesn't tell you why things are the way they are — the non-obvious invariants, the patterns that emerge from reading 50 files, the bugs that were fixed by changing two lines in a place that looks completely unrelated.

That's what Part 2 addresses: a note store where an AI editor can save findings in structured, tagged form — "the lock acquisition logic is at resolver.rs:214, and it acquires an exclusive file lock using fcntl.flock, not a threading primitive" — and retrieve them in under 50ms at the start of any future session. When /compact runs and replaces the conversation with a summary, exact signatures and line numbers evaporate — but notes don't. The indexer tells you where to look. The working memory layer tells you what you already know about what you found.

Summary of core decisions

Decision	Rationale
AST-aware chunking via tree-sitter	Complete functions as the unit of meaning. Biggest quality improvement over naive line windows.
Local embeddings (arctic-embed-m)	No API cost, no data leaving the machine. One-time 440MB download.
Hybrid BM25 + vector search	Concept queries route to vector. Exact symbol names route to BM25.
Symbol graph	Definitions, call edges, import edges, HTTP routes — exact graph traversal for questions text search cannot answer.
Six fallback strategies in vectr_locate	Exact → suffix → same_module → unique_name → import_chain → fuzzy. Each result carries its resolution strategy.
mtime cache + watchdog	Sub-5-second re-indexing on subsequent runs. In-session saves trigger background re-indexing automatically.

LLM Context Window Token Budget: Why Your Window Fills Up Fast

Swapnanil Saha — Tue, 26 May 2026 19:59:18 +0000

You build something with GPT-4o. The model supports 128,000 tokens. You think: that's enough for a full novel. Then, four or five conversation turns in, the model starts forgetting things that were said earlier. Eight turns in, you hit an error. You check the token count — you've used over 100,000 tokens, and you've typed maybe 400 words.

This isn't a bug. It's the predictable consequence of not accounting for where those tokens actually go. A context window isn't blank space waiting to be filled with your words. By the time the first user message arrives, it is already partially consumed — by system instructions, by tool definitions, by retrieved documents, by the tokens the model itself generated in earlier turns. In a production AI agent, 30–60% of the context window is gone before a user types anything.

What follows is a precise accounting of where those tokens go — the four layers that consume the window before users say anything, why the effective limit is substantially lower than the advertised one, what happens to response quality as the window approaches capacity, and which engineering patterns actually manage it at production scale.

Part 1: The Problem

1. The Illusion of Abundance

GPT-4o supports 128K tokens. Claude 3.5 supports 200K. Gemini 1.5 Pro has been demonstrated at a million tokens — roughly 750,000 words, about ten average novels. The numbers sound absurdly generous. How could you possibly run out?

Start with a calibration exercise. What is 128,000 tokens, actually?

In English prose, one token is roughly four characters — about three-quarters of a word. A 1,000-word article runs to around 1,300 tokens, so 128K tokens can hold close to 96,000 words of clean text. That genuinely is a lot.

But text in an LLM application is rarely clean English prose. It is JSON payloads from tool calls. It is API responses full of structured data. It is code. It is URLs. It is conversation history with speaker labels, timestamps, and formatting. All of these serialize into tokens at rates much higher than 4 characters per token.

Then there is the question of performance. The advertised number represents a technical limit — the longest sequence the model can physically process. It does not represent the length at which the model operates at peak accuracy. Research has repeatedly found a significant gap between the two. Long-context benchmarks like RULER (2024) and HELMET (2024) found that in adversarial multi-document tasks, most frontier LLMs showed accuracy drops well before 32K tokens — GPT-4o fell from near-perfect baseline scores to the high-60s percentage range at 32K in some configurations. The technical limit says 128K. The accuracy cliff arrives much earlier.

The Effective Limit Is Not the Advertised Limit
Models claiming 200K context windows show measurable quality degradation around 130K tokens in practice. Treating the advertised number as your operating budget is how production systems quietly degrade without triggering any explicit error.

Cost is the third angle. Every token in the context is a token billed. At GPT-4o's pricing, 128K tokens of input costs several dollars per call — and agents often make dozens of calls per session, each with the full accumulated context. The monthly bill from a badly-managed context window can surprise you well before any error appears in the logs.

2. How Tokens Are Counted — and Why the Count Surprises You

An LLM does not read text. It reads a sequence of integers. Before any word reaches the model, it passes through a tokenizer that converts characters into integer IDs from a vocabulary of roughly 50,000–200,000 entries. The tokenizer used by GPT-4 and GPT-4o is called cl100k_base; it has about 100,000 vocabulary entries. OpenAI's newer models use o200k_base, with about 200,000.

The vocabulary is built using BPE — Byte Pair Encoding. The name comes from the construction: you start with individual characters, then repeatedly merge the pair of adjacent symbols that appears most often in your training corpus, replacing each occurrence of that pair with a new combined token. Do this enough times and common English words end up as single tokens. The algorithm learns what to merge entirely from what was common in the training text — mostly English prose on the internet. That's why "the", "is", "running" each become a single token, while "tokenization" becomes ["token", "ization"] — less common as a whole word, so BPE never fully merged it. Characters and raw bytes are the fallback for anything the vocabulary doesn't cover. The consequence is simple: anything that wasn't well-represented in training data — JSON brackets, URL slashes, code indentation — never got merged aggressively, so those sequences remain expensive in tokens relative to the characters they contain.

The rule-of-thumb of 1 token ≈ 4 characters holds for clean English prose — decent enough for napkin estimates. It falls apart under several conditions that appear constantly in real applications:

Numbers tokenize unexpectedly. BPE learns tokens from frequency in training data. The number "2023" is common in training data — it became a single token. But "2026" is less common, and "19847" is rare — these get split into per-digit or per-pair tokens. The price "USD 1,234,567.89" produces approximately 10–12 tokens, because the commas, period, digits, and currency symbol may each claim separate tokens.

URLs are disproportionately expensive. A URL like https://api.example.com/v2/users/12345 looks compact — 38 characters, which by the prose rule should be about 9–10 tokens. In practice it is closer to 15–20 tokens. Slashes, dots, hyphens, underscores, and alphanumeric path segments each claim their own tokens or merge into small fragments, because URLs are structurally uncommon in prose.

JSON and structured data use roughly 2x the token count of plain text. Consider:

Plain text: The user's name is Alice, she is 28 years old, and her account is active.
JSON:       {"user": {"name": "Alice", "age": 28, "status": "active"}}

The plain text version: approximately 18 tokens. The JSON version: approximately 22 tokens — and this is a trivially small object. Real API responses with deeply nested keys, repeated field names, and verbose formatting can be far more expensive. Every brace, colon, and comma is a token or part of a token. A 500-word JSON payload can use 800+ tokens.

Code tokenizes inefficiently in some languages. Research found that Python uses roughly 46% more tokens than equivalent Haskell to express the same computational idea. Python's indentation-based structure requires whitespace tokens, and Python's identifiers were less densely represented in the pre-GPT-4 training corpora.

Analogy: The Luggage Weight Problem
Think of the context window as checked baggage with a weight limit, not a size limit. A suitcase full of dense sweaters weighs less than one with foam packing material filling the same volume. Plain prose is the dense sweaters — you pack a lot of meaning into few tokens. JSON, URLs, and code are the foam — structurally bulky, meaning-sparse, yet they count toward the same limit.

Part 2: The Consumers

3. The Four Layers That Eat Your Context Window

Every LLM API call is a full context payload assembled from four distinct layers. Most developers think about only one: the user's current message. The other three arrive already loaded — silent costs that accumulate before the user types anything.

Layer 1: The System Prompt

The system prompt is the foundational layer. It is always present, on every API call. A minimal system prompt — "You are a helpful assistant" — costs about 7 tokens. But real production system prompts are not minimal.

A typical customer-facing chatbot system prompt contains: the model's persona and tone guidelines, a list of topics it should and should not address, instructions about response format, domain-specific knowledge, legal disclaimers, and formatting instructions. Measured in practice, these range from 800 to 2,500 tokens. They are charged on every single API call. A 1,500-token system prompt running 1,000 calls per day costs you 1.5 million input tokens per day before a user says anything.

Layer 2: Tool Schemas

When you give an LLM access to external tools, you must describe each tool to the model in the context window. These descriptions are written in JSON and can be verbose. A single moderately documented tool schema costs roughly 200 tokens. An agent with five tools carries around 1,000 tokens of tool descriptions on every call, before any user input. The JSON structure alone — all those braces, colons, and quoted keys — is part of why the token cost is higher than reading the description would suggest.

Layer 3: Retrieved Context (RAG)

Many production LLM applications retrieve relevant documents from a database and inject them as supporting material. A typical RAG retrieval returns 3–8 document chunks, each 300–600 tokens. Three chunks at 400 tokens each: 1,200 tokens. Eight chunks at 500 tokens each: 4,000 tokens. In a research assistant with a generous retrieval budget, you might inject 8,000–12,000 tokens of context per query.

The Hidden Fixed Cost
System prompt + tool schemas is your fixed cost floor. It doesn't change turn-to-turn. It can easily reach 2,000–4,000 tokens in a real agent — charged on every single API call in your fleet.

Layer 4: Conversation History

The model has no persistent memory. You create the illusion of memory by re-sending the full conversation history on every API call. Every turn appends two new entries (a user message and a model response) to a history that is re-sent in its entirety. Model responses can be long — a detailed answer with a code snippet might be 600–800 tokens. After ten exchanges, the conversation history alone can be 8,000–12,000 tokens.

4. Context Creep — Watching the Window Fill

The process by which a context window fills over a conversation has a name in production systems: context creep. Consider a realistic customer support agent: 1,200-token system prompt, three tool schemas totaling 600 tokens, RAG retrieval returning two chunks (~800 tokens per turn), user messages averaging 60 tokens, model responses averaging 350 tokens.

Context budget:
  Fixed overhead: 1,200 + 600 = 1,800 tokens
  Per-turn RAG:   800 tokens
  Per-turn history growth: 60 (user) + 350 (model) = 410 tokens

  Turns until 80% of 128K:
    (1,800 + n × 800 + n × 410) ≥ 102,400
    n × 1,210 ≥ 100,600
    n ≈ 84 turns

  If model reply averages 800 tokens instead:
    Per-turn growth: 60 + 800 = 860
    n × 1,660 ≥ 100,600
    n ≈ 60 turns

Change the model reply length to 800 tokens — a detailed-answer agent — and the window hits 80% around turn 60 rather than 84. Quality degradation begins before you hit the hard limit.

Part 3: The Physics

5. KV Cache Memory — Why Context Has a Physical Cost

The context window limit is not an arbitrary policy. It is enforced by physics — GPU memory.

The transformer's attention mechanism works by comparing every token in the context with every other token. For each token, the model creates a query ("what am I looking for?"), and every other token offers a key ("what do I contain?"). A third vector — the value — carries the actual information that gets passed when attention is high. Assembled across all tokens, these become the matrices Q, K, and V:

Attention(Q, K, V) = softmax(QKᵀ / √d_k) · V

The QKᵀ product is an n × n matrix where n is the sequence length. Doubling n quadruples this computation.

There are two distinct computational phases in LLM inference. Prefill processes the entire input prompt at once — O(n²) per attention layer. Implementations like FlashAttention reduce the memory bandwidth pressure dramatically via tiled computation, but the asymptotic complexity doesn't change. Decode generates one token at a time, attending only to the current token against the cached history — O(n) per step with the KV cache. Without caching, decode would also be O(n²). The KV cache converts decode from O(n²) to O(n) at the cost of memory.

KV Cache Memory Formula (Multi-Head Attention):

KV_memory = 2 × n_layers × n_heads × d_head × seq_len × bytes_per_param

For a 7B-parameter model with standard MHA (32 layers, 32 heads, head_dim 128) at bfloat16 (2 bytes):

KV_memory per token ≈ 2 × 32 × 32 × 128 × 1 × 2 = 524,288 bytes ≈ 0.5 MB

At 128K context: 0.5 MB × 128,000 = 64 GB of KV cache alone — more than the model weights at bfloat16 (~14 GB).

Note on GQA and MLA: Most modern models (Llama 3, Mistral, GPT-4o) use Grouped-Query Attention (GQA), which reduces the KV cache by sharing key-value heads across groups of query heads. A model with 32 query heads and 8 KV heads (4× reduction) brings the per-token cache from ~0.5 MB to ~0.125 MB — about 16 GB at 128K context. Still the dominant memory consumer at long contexts. DeepSeek-class models use Multi-head Latent Attention (MLA), which compresses the K and V projections into a low-rank latent space before storing them, achieving 5–10× memory reduction over standard MHA.

A 70B MHA model (80 layers, 64 heads, head_dim 128, bfloat16) runs to roughly 2.5 MB per token: 2 × 80 × 64 × 128 × 2 bytes = 2,621,440 bytes. At 128K context that's ~320 GB — which is why providers either cap context length aggressively for large models, or charge steeply for long-context calls. GQA with 8 KV heads drops it to ~40 GB, still substantial.

Prompt caching (available from OpenAI, Anthropic, Google) caches the computed KV activations for repeated prompt prefixes. Subsequent calls beginning with the same prefix pay 50–75% less for those cached tokens and benefit from lower latency because the prefill phase for the cached portion is skipped. A stable system prompt is an ideal caching candidate. One practical constraint: both OpenAI and Anthropic require a minimum prefix length of at least 1,024 tokens before caching activates. A 200-token system prompt won't benefit — another reason to consolidate instructions into one substantial block rather than spreading them across multiple small messages.

KV cache quantization is an active area of production optimization: storing the K and V tensors in lower-precision formats (int8 or int4) cuts KV cache memory by 2–4× with modest accuracy penalties. Research like KVQuant explores going to 2-bit precision for certain layers while targeting 10M-token contexts on commodity hardware.

6. Lost in the Middle — Why Performance Collapses Before You Hit the Limit

Memory is the first constraint. Attention quality is the second — and it bites you even when your window is half-empty.

In 2023, researchers at Stanford and UC Berkeley published "Lost in the Middle." They gave LLMs a task requiring them to find a specific document from a set of twenty documents, all injected into the context window. The position of the relevant document was varied systematically.

When the relevant document was first or last, models retrieved it accurately. When it was in the middle positions, accuracy dropped by more than 30%. Newer models — Claude 3.5, GPT-4o — have partially mitigated this bias through long-context fine-tuning. "Partially" is doing a lot of work there: independent evaluations continue to find meaningful position-dependent performance gaps in all current models, even at lengths well within their advertised limits.

Analogy: The Lecture Hall Effect
Students reliably remember a lecture's opening and closing. What happened in the middle of hour one is murky. LLMs have an analogous concentration pattern: strong attention to the beginning and end of the context, with a trough in the middle.

The mechanism is structural. RoPE (Rotary Position Embedding), used in most modern architectures, encodes position as a rotation applied to query and key vectors. The mathematical property of this rotation is that the similarity score between two vectors naturally decreases as the distance between their positions increases. At short contexts, the decay is a feature. At long contexts, it becomes a bug: tokens in the middle of a 100K-token window are thousands of positions away from both the beginning and from where the model is currently generating, so their similarity scores are systematically suppressed.

A separate effect, context dilution, compounds this: longer surrounding irrelevant context degrades performance even when the relevant content is guaranteed present. The model's attention distributes across noise, reducing effective attention for the signal — like finding one red marble in a bag of ten thousand, even knowing it's there.

A Subtle RAG Bug
If your RAG system retrieves 8 documents and inserts them in the middle of a long conversation history, the most relevant chunks may be in the attention trough. The model generates a response, you see no error, but the answer doesn't reflect those documents. The failure is silent.

Part 4: Solutions

7. Token Budget Math — Calculating Your Real Available Space

Every LLM application needs an explicit token budget with five zones:

Zone	Typical token range	Fixed or variable?
System Prompt	500–2,500	Fixed per application
Tool Schemas	200–400 per tool	Fixed per agent
RAG Context	0–12,000	Variable per turn
Conversation History	0 → grows	Grows each turn
Generation Reserve	500–2,000	Reserved explicitly

The generation reserve must be reserved explicitly — if your prompt consumes the entire window, the model either generates nothing or truncates its response.

A worked example. Customer support agent, GPT-4o (128K):

Total window:          128,000 tokens
System prompt:          -1,400 tokens  (measured)
Tool schemas (4 tools):   -800 tokens  (measured)
Generation reserve:     -1,500 tokens  (set by us)
─────────────────────────────────────────
Available for dynamic:  124,300 tokens

  Of that:
    RAG budget:           20,000 tokens  (5 chunks × 4,000 avg)
    History budget:       ~104,300 tokens (fills over time)

  ─────────────────────────────────────────
  Turns until 80% full:
    80% of 128K = 102,400 prompt tokens
    Fixed overhead = 1,400 + 800 = 2,200
    Per-turn RAG = 800
    Per-turn growth = user avg (60) + model avg (350) = 410
    Turns until (2,200 + n × 800 + n × 410) ≥ 102,400
    n × 1,210 ≥ 100,200
    n ≈ 82 turns

82 turns sounds comfortable. But this assumes constant 350-token model replies. A user who triggers several detailed answers can double the history growth rate, cutting that to ~41 turns before the 80% threshold.

Measure, Don't Estimate
The system prompt and tool schema token counts must be measured with the actual tokenizer, not estimated from character counts. Log prompt_tokens and completion_tokens from every API response. The distribution of prompt_tokens over time is your context growth curve.

8. Four Strategies for Managing Context Window Limits

Strategy 1: Sliding Window

Keep only the most recent turns of conversation verbatim. In production, truncate by token count, not turn count — a 5-turn history could range from 500 to 8,000 tokens depending on response lengths.

# Turn-count version — simple, good enough for prototyping
MAX_HISTORY_TURNS = 20

def build_messages(system_prompt, history, new_message, rag_chunks):
    trimmed_history = history[-MAX_HISTORY_TURNS:]
    messages = [{"role": "system", "content": system_prompt}]
    if rag_chunks:
        context_block = "\n\n".join(rag_chunks)
        messages.append({"role": "system", "content": f"Context:\n{context_block}"})
    messages.extend(trimmed_history)
    messages.append({"role": "user", "content": new_message})
    return messages

# Production version — truncate by token count, not turn count
# HISTORY_TOKEN_BUDGET = context_limit - fixed_costs - generation_reserve
# Example for 128K window: 128000 - 2200 (sys+tools) - 1500 (reserve) - 20000 (RAG) ≈ 104000
HISTORY_TOKEN_BUDGET = 40_000  # adjust for your application

def build_messages_token_bounded(system_prompt, history, new_message, rag_chunks):
    fixed_tokens = count_tokens(system_prompt) + sum(count_tokens(c) for c in rag_chunks)
    new_msg_tokens = count_tokens(new_message)
    remaining = HISTORY_TOKEN_BUDGET - fixed_tokens - new_msg_tokens

    # Walk history from newest to oldest, keep what fits
    trimmed_rev = []
    for turn in reversed(history):
        turn_tokens = count_tokens(turn["content"])
        if remaining - turn_tokens < 0:
            break
        trimmed_rev.append(turn)
        remaining -= turn_tokens
    trimmed = list(reversed(trimmed_rev))

    messages = [{"role": "system", "content": system_prompt}]
    if rag_chunks:
        messages.append({"role": "system", "content": "Context:\n" + "\n\n".join(rag_chunks)})
    messages.extend(trimmed)
    messages.append({"role": "user", "content": new_message})
    return messages

The drawback of the sliding window is abrupt forgetting: when turn 1 drops, any fact established there is simply gone. For short-lived task-completion agents, this is fine. For long-running conversational assistants, it creates visible gaps.

Strategy 2: Hierarchical Summarization

Keep recent turns verbatim; compress older turns into a rolling summary.

async def maybe_compress_history(history, summary, buffer_size=10):
    verbatim_turns = history[-buffer_size:]
    turns_to_summarize = history[:-buffer_size]

    if not turns_to_summarize:
        return history, summary

    new_summary = await llm.complete(
        f"Existing summary: {summary}\n\n"
        f"New exchanges to incorporate:\n{format_turns(turns_to_summarize)}\n\n"
        "Update the summary to include these exchanges. "
        "Preserve all concrete facts, decisions, and commitments. "
        "Drop conversational filler. Be dense. Max ~400 tokens."
    )
    return verbatim_turns, new_summary

Cap the summary at 200–400 tokens. Run summarization asynchronously — don't make the user wait for the compression cycle.

Strategy 3: Token Compression (LLMLingua)

Use a compression model to identify and remove low-entropy tokens from prompts, achieving 2–3× compression with minor accuracy loss. The most effective targets are verbose system prompts, RAG context chunks, and few-shot examples.

Never apply compression to the current user message — compressing user input changes their meaning before the model sees it. Test in your specific domain for tasks where precision matters (legal, medical, code).

Strategy 4: Embedding-based Retrieval Over History

Store each conversation turn as a dense vector. At each new turn, embed the current user message and retrieve the most relevant prior turns by similarity. Concretely: as each turn completes, embed the user + assistant text and store it in a vector store alongside the full text. On the next user message, embed it, search for top-k similar turns, inject those into context. Keep only 2–3 verbatim recent turns for coherence.

The effect: only the conversation history relevant to the current question enters the context window. A user asking "what was the budget we discussed?" triggers retrieval of those turns — even if they happened fifty exchanges ago. This requires an embedding model, a vector store, and a retrieval call per user message (adding roughly 50–150ms round-trip with a managed API, under 10ms with a self-hosted model).

The four strategies are not mutually exclusive. Production systems often combine them: a sliding window of 5–8 verbatim turns + rolling summary + retrieval from older history covers all distance scales simultaneously.

9. The Practical Playbook

Short task-completion agents (under 20 turns): Use a sliding window of 10–15 turns. Reserve optimization effort for fixed-cost reduction: audit your system prompt for redundant language, consider dynamic tool registration (load only the tools relevant to the current turn).

Long-running conversational assistants: Implement hierarchical summarization with 8–12 verbatim turns. Cap summaries at 400 tokens. Run asynchronously. Periodically audit system prompt size — prompt creep through edits is real. A prompt that started at 600 tokens can quietly grow to 3,000 across six months of product changes.

Document-heavy research assistants (heavy RAG): Limit retrieval to 3–5 top chunks. Apply token compression to chunks before injection. Sort retrieved chunks so the most relevant appears last in the injected block — adjacent to the user question, within the recency attention peak.

Production agents with many tools: Use dynamic tool registration. A routing classifier (even a keyword matcher) identifies which tools are needed before the main model call and includes only those schemas — reducing 2,000 tokens of tool overhead to ~400 on most turns.

Context ordering (exploit the attention curve): Instead of the framework default (system → history → RAG → user), use: system → recent history (most-recent last) → RAG chunks (most relevant last, adjacent to the user message) → current user message. The most relevant content sits at the end of the context, within the recency attention peak. Older history — the least relevant content — occupies the lower-attention middle.

What to monitor:

prompt_tokens / context_limit — alert above 70%, act above 80%
Token count by zone per call — when total grows, know which zone is responsible
Quality signals segmented by context utilization — you may find degradation starts at 60% in your application

Conclusion: The Window Is a System Resource

A context window isn't a document store you fill until it overflows. It's a compute and memory resource with hard physical limits, a quality curve that degrades well before those limits, and an inference cost that grows with every token you put in it.

In a typical agent, the window is 30–60% consumed before the first user message lands. The fix isn't a bigger context window, though headroom helps. It's building a real budget: measure each zone with an actual tokenizer, set hard limits per zone, implement a context manager that enforces those limits on every call, and track utilization in production dashboards the same way you'd track memory or CPU.

The attention degradation problem — "lost in the middle" — adds a second dimension: even when your window is not full, quality depends on where in the window the important information sits. The primacy bias and recency bias are real, measurable effects that application design can exploit or fall victim to.

The four strategies aren't competitors — most production systems end up combining them. Sliding window for the recent turns, rolling summary for the older ones, compression for the RAG chunks, and retrieval for anything that needs to survive beyond the window. Start with the simplest thing that doesn't break your use case, and add layers as your traffic and conversation length grow.

Context engineering doesn't have the glamour of prompt engineering, but it's where most production LLM failures actually live. Missed retrievals, incoherent multi-turn conversations, bloated inference bills — these trace back to context mismanagement more often than they trace back to the wrong model. It fails silently, which is exactly why it's easy to ignore until you can't.

References

Research Papers

Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" — Stanford / Berkeley / Samaya AI, 2023. The original paper quantifying the U-shaped attention bias across context positions.
"Found in the Middle: Calibrating Positional Attention Bias" — 2024. Proposes an architectural fix to the lost-in-the-middle problem, recovering up to 15pp accuracy.
"Context Length Alone Hurts LLM Performance Despite Perfect Retrieval" — 2025. Demonstrates context dilution: longer irrelevant context degrades performance even when the relevant content is guaranteed present.
"KVQuant: Towards 10M Context Length LLM Inference with KV Cache Quantization" — 2024. Explores per-channel quantization of the KV cache to enable extreme context lengths on commodity hardware.

Technical References

OpenAI — Managing Conversation State — Official docs on conversation history management and token counting.
Anthropic — Context Window Documentation — Claude context limits, caching strategies, and best practices.
LLMLingua — Prompt Compression — Microsoft Research open-source project for token-level prompt compression.
KV Cache Memory: Calculating GPU Requirements for LLM Inference — Interactive calculator for KV cache memory requirements given model architecture parameters.

Background Reading

The Hidden Costs of Context: Managing Token Budgets in Production LLM Systems — TianPan.co, 2025. Production-focused survey of context management challenges.
Context Window Management for LLM Apps: Developer Guide — Redis, 2025. Practical implementation patterns for context management in production.
The Complete Guide to Text Embeddings, Vector Databases & LLMs — Swapnanil Saha, 2026. Deep background on tokenization, BPE, transformer attention, and RAG pipelines referenced throughout this post.

Why AI Code Assistants Waste Context — and How RAG Fixes It

Swapnanil Saha — Tue, 26 May 2026 19:25:33 +0000

Open a large file in your AI code assistant and ask it to refactor a function buried three hundred lines down. Watch it confidently produce something plausible but wrong — using an interface that was deprecated last sprint, calling a helper that doesn't exist in this service, ignoring a constraint in the module-level docstring that it technically "saw." The model didn't forget. The information was technically present in the prompt, but the transformer's attention mechanism never meaningfully focused on it. That's a different kind of failure, and it doesn't get better with a bigger context window.

There's a persistent intuition in this industry that more context is always better. Send the whole file. Send the whole codebase. This intuition breaks in a specific and measurable way. The mechanism is called attention dilution — softmax normalization means that every token in the context competes for a fixed budget of attention weight, and as the sequence grows longer, any given piece of information gets a smaller share of that budget.

This post walks through the transformer attention math to explain exactly why the naive approach fails, then covers how RAG (Retrieval-Augmented Generation) addresses it — by retrieving only the specific code chunks relevant to the current task and injecting those into the context window instead of dumping everything.

Part 1: The Problem with Stuffing

1. The Naive Approach: Just Send Everything

The first instinct when building a code assistant is to send as much context as possible. Your project has a utility module? Include it. There's a shared type definitions file? Throw that in too. If the model's context window is 128,000 tokens, fill it to the brim — more information has to be better, right?

This is called context window stuffing. Three things go wrong with it, and each gets worse as the codebase grows. The first is attention dilution — the focus of this section. The second is position bias (Section 3). The third is raw cost (Section 4). To understand why these happen, you need a concrete model of how a transformer actually reads a prompt.

A transformer does not read a prompt sequentially, the way a human reads a page from left to right. Instead, it processes all tokens simultaneously, and every token attends to every other token in the sequence. The attention mechanism is the machine that computes how much each token should "look at" every other token when forming its representation.

The output of attention for a single token is a weighted average of all the other tokens' value vectors. The weights are computed by comparing the current token's query vector against every other token's key vector. When you add more tokens to the context, you are not adding more information to a receptive mind — you are adding more competitors for a fixed budget of attention weight.

Analogy: Imagine you are in a room full of people, all talking at once. You can only pay 100 percent of your attention total — it does not grow with the number of people. With 5 people in the room, each gets roughly 20% of your focus. With 500, each gets 0.2%. When the relevant person finally says something, their share of your attention has collapsed to noise. That is what happens to code buried in a long prompt.

2. Why Attention Dilutes: The Math

The attention mechanism was introduced in the paper "Attention Is All You Need" (Vaswani et al., 2017). Its core computation is:

Attention(Q, K, V) = softmax( QK^T / √d_k ) · V

Where:

Q — the query matrix (what each token is "asking for")
K — the key matrix (what each token "offers" for comparison)
V — the value matrix (the actual content passed forward if selected)
d_k — the dimension of the key vectors (scales to prevent extreme dot products)
softmax — converts a vector of raw scores into a probability distribution that sums to 1

The notation QK^T means: for each token, compute a dot product between its query vector and every other token's key vector. The dot product is large when two vectors point in the same direction (high relevance between the pair), and near zero when they are orthogonal (unrelated). Multiplying by the transposed key matrix K^T does all N×N such comparisons in a single matrix operation. The result is a matrix of raw relevance scores. Dividing by √d_k prevents those scores from becoming so large that softmax saturates.

The softmax step is the dilution mechanism. Because softmax always outputs a probability distribution — all values sum to exactly 1 — attention weights are a zero-sum resource. When there are N tokens in the context, the average attention weight is 1/N, regardless of what any individual token does. The total budget is fixed at 1.0.

This does not mean every token gets exactly equal attention — the model can still concentrate on a small subset if the dot-product scores separate those tokens sharply from the rest. Softmax is non-linear and can be quite aggressive when there is a large score gap between relevant and irrelevant tokens. But in a real codebase, that gap is rarely clean. Hundreds of unrelated function definitions produce hundreds of tokens with moderately non-zero dot products — they're not completely irrelevant, they just aren't what you need right now. These tokens collectively consume most of the softmax budget. The useful signal must compete against this crowd, and as N grows, the signal's share degrades continuously. It isn't a cliff; it's a steady erosion that compounds with each additional file you stuff in.

Key Insight: The context window limit is not just a practical engineering constraint — it reflects a genuine quality degradation. The problem is not that the model cannot read long inputs. It is that as context grows, every individual piece of information receives proportionally less attention weight. More input does not mean more comprehension; it means each fact competes harder for finite attentional resources.

3. Lost in the Middle: Position Bias

Attention dilution is one problem. A second, independent problem compounds it: position bias. Modern language models do not attend to all positions in their context with equal reliability. They preferentially attend to tokens at the beginning and end of the sequence, and perform significantly worse on information placed in the middle.

This phenomenon was studied in a 2023 paper by Nelson Liu et al. titled Lost in the Middle: How Language Models Use Long Contexts. The researchers tested models on multi-document question answering, varying the position of the document containing the answer. When the answer document was at position 1 or last, accuracy was high. When it was at position 10 of 20 documents, accuracy dropped by more than 30 percentage points — even though the information was technically within the model's context window.

Two mechanisms contribute. The first is RoPE (Rotary Position Embeddings), the positional encoding scheme in most modern open-source language models (LLaMA, Mistral, GPT-NeoX). RoPE encodes position by rotating the query and key vectors by angles proportional to their positions. The dot product between a query at position m and a key at position n includes a term that decays with relative distance (m−n) — semantically relevant tokens far from the query position must overcome a rotational penalty to receive attention weight. Tokens near the start of the sequence are close to almost every other position, giving them a structural advantage.

The second mechanism is causal training recency bias. Language models are trained to predict the next token given all previous tokens. This reward signal pushes models to weight recent tokens heavily — the immediately preceding context is almost always the most relevant signal for next-token prediction during training. The middle of a long context rarely dominated training gradients, so models systematically underweight it. This effect was documented in GPT-3.5 era models well before RoPE became standard — it isn't purely an artifact of positional encoding, it's baked into causal pretraining. Both effects run in the same direction: the middle of a long context is structurally disadvantaged.

A 2024 paper from UW, MIT, and Google (Found in the Middle) demonstrated that this bias can be partially corrected by calibrating attention weights at inference time — but this requires modifying the model's internals, which is not available when calling an API.

Common Mistake: Many teams inject retrieved chunks at the end of the prompt, after a long system prompt and conversation history. This lands retrieved content in a position that gets the worst of both worlds: far from the beginning (losing the primacy advantage) and not at the very end (which is reserved for the generation target itself). The safest placement for retrieved code context is immediately before the user's specific question, near the end but not buried in the middle of a long history.

4. The Quadratic Cost Problem

Even if you were willing to accept degraded attention quality, there is a third reason not to stuff context: the compute cost of attention scales quadratically with sequence length.

To compute the full attention matrix, the model must compare every token's query against every other token's key. If your sequence has N tokens, this requires N × N comparisons. Doubling the context length quadruples the compute required for attention.

Time complexity of full self-attention: O(N² · d)

A 4× increase in context length → 16× increase in attention compute. A 10× increase → 100×.

FlashAttention (Dao et al., 2022) improves the memory profile to O(N) via tiling — it never writes the full N×N matrix to GPU memory. But the number of floating-point operations is still O(N²). Latency and cost still scale quadratically with sequence length.

In production, a code assistant filling 100,000 tokens of context is not just 10× slower than one filling 10,000 tokens — it is closer to 100× more expensive in attention compute alone. You are paying more to get worse results.

Part 2: How RAG Fixes It

5. RAG at a Glance: The Core Idea

Retrieval-Augmented Generation reframes the problem. Instead of asking "how can we give the model the whole codebase?", it asks: "how do we figure out which parts of the codebase are relevant to this specific completion request, and send only those?"

The answer has two phases. First, an offline indexing phase where the codebase is processed, divided into chunks, and each chunk is converted into a vector representation (an embedding) that captures its semantic meaning. These vectors are stored in an index optimized for fast similarity search. Second, an online retrieval phase that happens at query time: the developer's current context is converted into a query vector, and the most similar chunks from the index are retrieved and injected into the prompt.

The model then receives a context window that is not a random cross-section of the codebase — it is the small set of pieces most likely to be relevant to the task at hand.

The pipeline:

Parse & Chunk — split at function/class boundaries, not arbitrary token counts
Embed Chunks — convert each chunk to a vector with a code embedding model
Build Search Index — ANN index for dense retrieval + BM25 index for lexical retrieval
Embed the Query — convert current cursor context to a query vector
Retrieve Top-k — run hybrid search (dense + BM25), fuse results
Inject & Generate — inject top 3–5 chunks into the LLM prompt, immediately before the user's request

Steps 1–3 happen once (or on incremental file changes). Steps 4–6 happen on every completion request. The parts where most implementations go wrong: chunking (using fixed-size splits instead of AST boundaries), retrieval (using only dense search and missing exact identifier queries), and injection order (burying retrieved context in the middle of the prompt).

6. Chunking for Code: Why Fixed-Size Fails

Code has structure that text does not. A function is a unit of meaning. Fixed-size chunking — splitting every file every 256 tokens — splits in the middle of functions, destroying logical units.

Consider a Python function that is 80 lines long. With a 50-token chunk size, it gets split into chunks that look like:

Chunk A: def process_payment(order_id, amount, currency="USD"):
    """Process a payment..."""
    conn = get_db_connection()
    try:
        txn = conn.begin_transaction(

Chunk B:   order_id=order_id,
  amount=amount,
  currency=currency
)    except DatabaseError as e:
        log_error(e)
        raise PaymentError(str(e))

Neither chunk represents the function accurately. The embedding of Chunk A does not represent "a payment processing function" — it represents a truncated fragment.

AST-based chunking uses tree-sitter to parse each file and extract logical units at language-defined boundaries: function definitions, class bodies, method groups. Each chunk's metadata includes file path, start line, end line, and node type. This metadata is as important as the chunk text itself — it tells the retrieval system where in the codebase this chunk lives.

One practical addition: each chunk can be augmented with a small surrounding context for embedding purposes — the preceding import block, the class it belongs to, or the file's module-level docstring. This gives the embedding model enough context to produce a vector that reflects the chunk's role in the larger structure. The key is that this surrounding context is used only for embedding, not retrieved as part of the chunk text.

The overlap trap in code: Sliding window overlap (copying N tokens from one chunk into the next) is useful in prose. In code it often makes things worse: the overlap introduces duplicate logic into separate chunks, making embedding space crowded with near-identical vectors. For code, the recommended approach is to store a "parent context" chunk separately — always inject the enclosing class signature alongside any function chunk, rather than copying the previous function's body into the current chunk. The Continue open-source IDE extension uses this approach.

7. Retrieval Strategies: Dense, Sparse, and Why Code Needs Both

Dense retrieval converts query and each chunk to vectors, then finds the most similar by cosine similarity. It can match meaning even when exact words differ — "how do we handle rate limit errors?" surfaces functions named throttle_on_429 or backoff_retry.

The embedding model used matters significantly. Code-specialized models like voyage-code-3 — purpose-built for code retrieval, top-ranked on code retrieval benchmarks (2025) — produce substantially better representations for function bodies, type signatures, and API calls than general-purpose models. text-embedding-3-large is a strong general-purpose embedding model suited for mixed code + documentation retrieval, but it wasn't specifically designed around code.

BM25 (lexical/keyword retrieval) counts words. It excels at exact matches — a developer looking for PaymentGateway.process_refund will find it immediately. Error codes, configuration key names, and exact API method names are better retrieved lexically than semantically. For code, the asymmetry is important: queries for exact identifiers favor BM25. Queries for concepts and behaviors favor dense retrieval. The right system runs both.

8. Hybrid Search and Reciprocal Rank Fusion

Running both methods produces two ranked lists that need combining. BM25 scores and cosine similarity scores live in completely different numerical ranges — you cannot add them directly.

Reciprocal Rank Fusion (RRF) avoids the normalization problem entirely by ignoring raw scores and working only with ranks. The word "reciprocal" means 1/x — the score assigned to a document is the reciprocal of its rank in each list:

RRF_score(d) = Σ_{r ∈ R} 1 / (k + rank_r(d))

Where:

R = set of ranked lists (BM25 list, dense list)
rank_r(d) = position of document d in list r (1-indexed)
k = smoothing constant (default 60, from Cormack, Clarke & Buettcher 2009 — empirically robust across many retrieval tasks). Increasing k makes the formula more conservative, rewarding consistent mid-rank appearances over a single strong rank.
If a document does not appear in a list, its contribution from that list is 0

A document ranked #1 in both lists scores ≈ 0.033. A document ranked #1 in one list but #100 in the other scores ≈ 0.022. Candidates that both BM25 and semantic search agree on float to the top.

9. Reranking: The Final Sorting Pass

After hybrid search and RRF, you have ~20 candidate chunks. A cross-encoder reranker takes both the query and a candidate chunk as a single concatenated input and produces a relevance score. Because both texts pass through the model together, the model can attend to query-document relationships that a bi-encoder cannot — query and document never interact during bi-encoder encoding.

The practical architecture: use fast bi-encoder retrieval (dense + BM25 + RRF) to get the top 20 candidates, then run a cross-encoder on those 20 for final ordering. The top 5 go into the prompt.

Cross-encoder context window limits: Cross-encoders are themselves transformer models with context window limits. General-purpose reranker models like ms-marco-MiniLM-L-12-v2 support 512 subword tokens — which is often enough for a single short function, but not for large class bodies. For retrieval pipelines that surface larger chunks, use a reranker with a larger window: Cohere Rerank 3 supports 4,096 tokens; voyage-rerank-2 supports 16K. If the combined chunk+query still exceeds the limit, truncate the chunk from the bottom — the function signature and docstring are more informative for reranking than the implementation tail.

For code with strong AST chunking and a good code embedding model, hybrid bi-encoder retrieval is often sufficient for most queries. Reranking becomes most valuable when queries are ambiguous or when the codebase has many semantically similar functions. It adds 50–200ms of latency, so benchmark before committing.

Part 3: In Production

10. How Cursor Does It: A Reference Architecture

When you open a project in Cursor, it chunks local files and sends them to its servers, where they are embedded (via OpenAI's API or a custom model) and stored in Turbopuffer — its vector store of choice. File paths are obfuscated client-side before any data leaves your machine. Embeddings are cached by chunk hash, making incremental re-indexing fast.

At query time, Cursor monitors the active cursor position and constructs a composite signal: the current file's surrounding code, any open editor tabs, and recent edit history. This signal is embedded into a query vector, sent to Turbopuffer for ANN search, and the top-k results are retrieved. The actual code is read from local disk; the model only sees the retrieved text.

@Codebase in Cursor's chat is the explicit trigger for a full retrieval pass over the indexed codebase. Without it, Cursor uses a lighter heuristic based on open tabs and file imports. @Docs and @Web extend the same pipeline beyond the local codebase.

One important architectural note: the embedding model used to index the codebase is separate from the generative model used to produce completions. Cursor uses a lightweight, fast embedding model for indexing (optimized for latency and throughput over millions of chunks) and a larger, slower generative model for the actual completion. When building a similar system, these two components have independent optimization concerns — do not assume the same model serves both roles.

GitHub Copilot's context construction follows a similar pattern. For inline completion, it uses the current file content around the cursor plus a Jaccard similarity heuristic to find other open tabs that share significant token overlap with the current file. The @workspace symbol in VS Code triggers a more thorough indexing-based search, analogous to Cursor's @Codebase. Copilot's default inline completion mode is a fast, low-latency path that does not run full vector retrieval on every keystroke — full retrieval is reserved for explicit chat interactions.

11. Tradeoffs and Limits of Code RAG

Scenario	RAG Behavior	Mitigation
Cross-file dependency reasoning	Each retrieved chunk is a fragment; the model may not understand how three retrieved functions compose at the call site	Include file path + line range metadata; retrieve parent class or module-level imports alongside function bodies
Newly created files not yet indexed	Invisible to retrieval until the index is rebuilt	Incremental indexing on file-save events; maintain a pending index queue
Query is too vague	"fix the bug" → retrieves generic results	Use cursor position + surrounding error message as primary query signal
Minified or generated code	Lock files, protobuf generated code pollute the index	Maintain a .gitignore-style exclude list for the RAG indexer
Very large monorepos	Recall degrades; indexing is slow	Scope index to current working subdirectory or per-service sub-indices
Schema/type changes	Stale embeddings give the model outdated type signatures	Invalidate embeddings on file write by chunk content hash

Does a larger context window make RAG obsolete? As context windows grow to 1M and beyond — Llama 4 Scout hit 10M tokens in 2025, Gemini 1.5 Pro supported 1M — this question keeps coming up. The practical answer is no, though the reasoning matters. A 200,000-line Python codebase easily exceeds 2 million tokens. Most production monorepos are far larger. More importantly, the attention quality degradation described in Sections 2 and 3 doesn't disappear with a larger nominal window. Those long-context models achieve their range through techniques like NTK-aware RoPE scaling (which extends the effective frequency range of positional encodings) and sparse attention patterns (which skip computation on distant token pairs) — these help with extrapolation but don't eliminate the position bias at extremely long ranges. And practically: a 1M-token prompt is expensive and slow even on state-of-the-art hardware. For interactive code assistance, stuffing the full codebase is off the table regardless of window size.

Large context windows and RAG do different jobs. RAG decides what deserves to be in the context window. The context window determines how much you can fit once you've been selective. A well-tuned system retrieves the right 5,000 tokens from a 10M-token codebase and puts them in a 128K window with room left for conversation history and tool outputs.

12. Building Your Own Code RAG Pipeline

Parsing: Use tree-sitter with a recursive AST walk — iterating only over root_node.children misses deeply nested functions and class methods.

# Pseudo-code: AST chunk extraction with tree-sitter (v0.21+ API)
import tree_sitter_python as tspython
from tree_sitter import Language, Parser

PY_LANGUAGE = Language(tspython.language())
parser = Parser(PY_LANGUAGE)

TARGET_TYPES = {"function_definition", "class_definition"}

def walk_tree(node, source_code: str, file_path: str, chunks: list):
    """Recursively walk the AST to catch nested definitions
    (methods inside classes, functions inside functions, etc.)"""
    if node.type in TARGET_TYPES:
        chunk_text = source_code[node.start_byte:node.end_byte]
        chunks.append({
            "text": chunk_text,
            "file": file_path,
            "start_line": node.start_point[0],
            "end_line": node.end_point[0],
            "type": node.type
        })
        # For class_definition, continue recursing to capture methods.
        # For function_definition, stop — we want the whole function,
        # not its nested helpers as separate chunks.
        if node.type == "class_definition":
            for child in node.children:
                walk_tree(child, source_code, file_path, chunks)
    else:
        for child in node.children:
            walk_tree(child, source_code, file_path, chunks)

def extract_chunks(source_code: str, file_path: str) -> list[dict]:
    tree = parser.parse(source_code.encode())
    chunks = []
    walk_tree(tree.root_node, source_code, file_path, chunks)
    return chunks

Embedding models:

Model	Context window	Strengths	When to use
`voyage-code-3`	16K tokens	Purpose-built for code; top-ranked on code retrieval benchmarks (2025)	Production code assistant, maximum retrieval quality
`text-embedding-3-large`	8K tokens	Strong general performance; well-supported; large community	Mixed code + documentation retrieval; existing OpenAI integrations
`nomic-embed-code`	8K tokens	Open-weight; can run locally; no API cost	Air-gapped environments; cost-sensitive deployments; on-prem

Vector store:

pgvector in Postgres — sufficient for single-developer or small-team tools
Qdrant — supports both dense and sparse vectors in a single collection, enabling native hybrid search without maintaining two separate stores

Prompt injection template:

You are a coding assistant for this codebase.

## Relevant context from the codebase:

### [payments/gateway.py · lines 42–87]

python
{chunk_1_text}


### [payments/exceptions.py · lines 1–24]

python
{chunk_2_text}


### [payments/models.py · lines 88–112]

python
{chunk_3_text}


## Current task:
{user_request}

Include file path and line numbers in each chunk header. These cost very few tokens but give the model the module structure needed to generate correct imports and references.

Do not retrieve more than you need. It is tempting to inject 10–15 chunks to "give the model more information." Resist this. Each additional chunk increases context size (paying the quadratic cost from Section 4), increases attention dilution, and reduces the proportion of the context that is highly relevant. In practice, 3–5 high-quality chunks typically outperform 15 lower-quality ones. Invest in retrieval quality, not retrieval quantity.

The Through-Line

The surprising thing about attention dilution is that it isn't a bug you can patch. It's a structural property of softmax normalization — the total attention weight sums to 1.0 regardless of sequence length, so every token you add is competing with every other for a share of that budget. More context doesn't mean more understanding; it means each fact gets a smaller slice. The lost-in-the-middle position bias makes it worse: code injected into the middle of a long prompt is structurally disadvantaged by both RoPE's distance decay and the recency bias that causal pretraining instills. Knowing this changes how you think about the whole problem.

RAG doesn't solve attention dilution — it sidesteps it. Instead of sending everything and hoping the model finds what's relevant, it figures out what's relevant first and sends only that. The context window ends up containing what actually matters for the task: the right type definitions, the right helper functions, the right error handling patterns.

In practice: below roughly 3,000–5,000 lines, context stuffing usually works well enough. Above that, the problems stack up fast. At 50,000+ lines, naive stuffing reliably hurts. At 500,000+ lines, AST chunking, hybrid BM25 + dense retrieval, RRF fusion, and careful prompt injection aren't premature optimization — they're the baseline.

References

Foundational Papers

Vaswani et al. (2017) — Attention Is All You Need. NeurIPS. The original transformer paper introducing scaled dot-product attention.
Liu et al. (2023) — Lost in the Middle: How Language Models Use Long Contexts. Stanford / Berkeley. Empirical study of U-shaped attention bias and the 30% accuracy drop at mid-context positions.
He et al. (2024) — Found in the Middle: Calibrating Positional Attention Bias. UW / MIT / Google. Proposed calibration method that partially corrects RoPE position bias at inference time.
Survey (2025) — Retrieval-Augmented Code Generation: A Survey. Comprehensive survey of RAG approaches specifically for code generation and repository-level tasks.

RAG & Retrieval

Cormack, Clarke & Buettcher (2009) — Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods
Dao et al. (2022) — FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
RAG vs Large Context Window: Real Trade-offs for AI Apps — Redis Engineering Blog
Context Window Optimization: Why Ranking, Not Stuffing, Is the Scaling Law for Agents — Shaped AI
RAG for LLM Code Generation using AST-Based Chunking — Vishnudhat Natarajan
Better Retrieval Beats Better Models for Large Codebases — Stéphane Derosiaux

Code Assistants & Architecture

How Cursor Actually Indexes Your Codebase — Towards Data Science
How GitHub Copilot Works — Quastor Engineering
What is Retrieval-Augmented Generation? — GitHub Blog

Hybrid Search & Ranking

BM25 vs Dense Retrieval for RAG: What Actually Breaks in Production — Ranjan Kumar
Hybrid Search: BM25 and Dense Retrieval Combined — Michael Brenndoerfer

DEV Community: Swapnanil Saha

The Bugs Only Dogfooding Finds: A Month Living Inside My Own Memory Daemon

Part 1: The Setup

1. Living Inside the Thing You Are Building

2. Why a Green Test Suite Proves Less Than It Looks

Part 2: Six Bugs

3. The Remediation That Was the Hazard

4. The Version Check That Cried Wolf Backwards

5. The Restart That Orphaned the Session

6. The Mock That Lied Politely

7. The Test That Was Really Testing My Filesystem

8. The Laptop That Ate the Fleet

Part 3: The Pattern

9. Six Boundaries a Test Suite Cannot Cross

10. What Dogfooding Is Bad At

11. The Rule I Needed Telling Three Times

Summary

Related reading

Agent Memory Needs a Trust Ladder: Provenance, Revocation, and Notes That Lie

Part 1: The Problem With Two Verbs

1. Save and Search Is Not a Memory System

2. Three Ways a Stored Note Lies

1. Wrong when written

2. Right then stale

3. Misattributed authority

Part 2: What I Shipped

3. The Trust Ladder: Three Provenance Classes

The framing is the interface

The write-time enum is where the security lives

4. Promotion: Trust Is Earned, Never Asserted

5. Revocation Is Not Deletion

The retrieval property that makes it work

The correction and the revocation are one write

Revocation does not set a validity window

Revocations are themselves sometimes wrong

6. Staleness: A Changed Anchor Means May, Never Is

7. Delivery Rules: Trust Decides Where a Note Shows Up

Part 3: What I Have Not Solved

8. The Missing Middle Class

9. Mixed Provenance, and the Attestation You Cannot Delegate

The candidate design

10. The Banner Is Unmeasured

The experiment that would settle it

11. What I Would Build Next

Related reading

The Sentinel Pattern: A Multi-Agent Dev Loop That Doesn't Trust Its Own Agents

Part 01: The Shape

01. The Unit of Work Is Not the Agent

02. Three Roles, and What Each One Is Trusted For

The sentinel

The coders

The adversarial reviewers

03. The Ledger: One Append-Only File

Part 02: The Gate

04. Rule 1: Never Merge on a Claim, Enumerate the Artifacts

Turning a claim into evidence

05. What the Gate Costs: the Arithmetic of a Single Bottleneck

Two things the formula leaves out

Where your fleet stops paying

Part 03: Rules Written by Failures

06. Rule 2: Worktree Isolation, and the Trap Inside It

07. Rule 3: Tier the Models by Failure Cost

08. Rule 4: Killed Agents Get Resumed, Not Restarted

09. Rule 5: Quota Is a Shared Resource, So Everything Is Resumable

10. Rule 6: Laptop Physics Is Part of the Architecture

11. Rule 7: Notes Are the Handoff, Transcripts Are Not

Part 04: The Bill

12. Honest Accounting: What This Buys, What It Costs, When Not To

What it buys

What it costs

When not to use it

Summary

Related reading

The Agent Never Chooses to Remember: Memory as a Harness Property

The Agent Never Chooses to Remember: Memory as a Harness Property

Part 1: The Assumption Nobody States

01. One Assumption, Every Agent Memory Product

02. The Memory Tier You Do Not Choose

Part 2: What the Runs Actually Showed

03. Zero Voluntary Memory Operations in 114 Turns