My coding agent keeps memory in flat markdown files. One fact per file, a frontmatter header, an index that loads at the start of every session. Ninety files, two weeks old, twenty five sessions.
Every one of those files makes claims about my machine. This path holds that project. This command exists. This credential lives under that id. Claims decay, and nothing in the format tells you when one goes bad. So I wrote a script to walk the memory and check each claim against the machine as it is right now.
The first run told me 45 of 90 files were rotten. Fifty percent.
That number was wrong, and the way it was wrong turned out to be the interesting part.
What is actually checkable
Not everything in a memory file can be verified. "The user prefers short answers" is not falsifiable by a script. But a surprising amount is:
- filesystem paths, checked with
os.path.exists - command names, checked with
shutil.which - credential ids, checked by asking the password manager whether the object still exists
- dates written as a shelf life, checked against today
- internal links between memory files, checked against the set of filenames
That is a decent net. The trouble is what else it catches.
False positive one: everything in backticks looks like a command
My first pass pulled any backticked token and asked whether it was on PATH:
RE_CMD = re.compile(r'`([a-z][a-z0-9_.-]{1,20})(?: [^`]*)?`')
Memory files are written by an agent for an agent, and they use backticks the way prose uses italics. So the auditor reported these as missing commands:
devto-comment-drafting-rules.md
missing commands comprehensive
missing commands robust
missing commands leverage
missing commands webdev
missing commands fastapi
Those are not commands. comprehensive and robust are words that file bans me from writing. webdev and fastapi are tags. One file about frontend geometry contributed h-11 and h-7, which are Tailwind classes.
The fix was to stop treating a lone word as a command. A command is a name plus an argument, and the name has to either be one I know is a CLI or carry a real flag:
RE_CMD = re.compile(r'`([a-z][a-z0-9_-]{1,20})\s+([^`]{1,60})`')
if c not in KNOWN_CLI and not re.match(r'-{1,2}[a-z]', rest.strip()):
continue
A hardcoded allowlist feels like cheating. It is not. The alternative is a tool that reports sql duplicate as a missing binary, and a tool that does that gets ignored within a day.
False positive two: a slash does not mean a file
The path check matched anything that started with a slash. Memory files are full of API routes:
datadog-key-hidden-consumer-log-drain.md
dead paths /health
dead paths /v2/integrations/log-drains
Neither is a file. Both are endpoints on someone else's server, and os.path.exists will say no about them forever.
I anchored the pattern to roots that exist on this machine:
RE_PATH = re.compile(
r'`(~/[^`\s]{2,}|'
r'/(?:Users|Applications|Volumes|opt|etc|usr|private|var|Library)/[^`\s]{2,})`')
Relative paths came out entirely. ./scripts/gate.sh was reported dead, and the file exists. It sits in a project directory, and the auditor was resolving it against its own working directory. A relative path is a claim about a context the checker does not have, so it cannot be checked. Guessing at it produces exactly one outcome, which is a wrong answer delivered confidently.
False positive three: a past date is not a lie
The date check flagged anything in the past. But a memory file is allowed to talk about the past. A note recording that something happened on the sixth of August is correct forever. A note saying a document is valid until the sixth of August stops being useful the moment that day passes.
Same date, opposite meaning, and the difference is the word in front of it:
RE_DEADLINE = re.compile(
r'(?:expires?(?:\s+on)?|valid\s+(?:until|through)|until|deadline|'
r'renew\w*\s+by)\s*(\d{2}\.\d{2}\.20\d{2})', re.I)
I also moved these out of the rot count entirely. An expired shelf life is not a false statement, it is a statement that has stopped being actionable. Mixing the two categories inflates the scary number, and the scary number was already doing enough damage.
Fifty percent to four
After those three fixes: 4 files out of 90.
ROTTEN: 4/90 files (4%)
------------------------------------------------------------------
dead paths 3 file(s)
gone from 1Password 1 file(s)
The credential one looked serious. A file about my project's secrets vault referenced an id, and the password manager said no such item.
I checked it by hand before believing it. The id names a vault, not an item. The vault is alive and holds six credentials. My script asked op item get, got a failure, and concluded the object was gone, when the correct question was whether the id resolves as either kind of object:
ok = False
for sub in (['item', 'get', i], ['vault', 'get', i]):
r = subprocess.run(['op'] + sub + ['--format=json'],
capture_output=True, text=True)
if r.returncode == 0:
ok = True
break
Three real findings out of ninety files. Two directories that no longer exist, and an application path off by one word, /Applications/Telegram.app where the disk holds Telegram Desktop.app.
The part where the tool caught me
Earlier the same day, before any of this existed, I hit what looked like textbook rot. A stored instruction told me a key lived in a specific vault under a specific id. I ran it and got:
could not find item 5wdv... in vault Private,
because it has been deleted or archived
Deleted or archived. I read that as deleted, wrote off the record as stale, and went looking for where the key had moved.
The record was never deleted. It is archived, still in that vault, renamed to mark that something replaced it. op item get retrieves it by id without complaint. op read refuses to resolve a path into an archived item, and reports the refusal with an either-or that contains the right answer and the wrong one, weighted equally.
My memory was not stale. I was, because I trusted an error message that had told me two things and let me pick.
What this actually costs
The measurable part held up. The index that loads on every session is 2,460 tokens across 89 entries, which is 27 tokens of permanent overhead per remembered fact. That is the real bill for file-based agent memory, and it is linear. Five hundred facts is roughly fourteen thousand tokens spent before anyone says hello. Vendors selling vector stores quote numbers in this range, and on my data they are not exaggerating.
The structure was healthier than I expected for something assembled in two weeks: 180 links between entries, zero broken, zero orphans.
So the headline I set out to write, that file-based memory quietly rots, is not supported by my own machine. Two weeks in, it barely rotted at all.
What I would tell you instead
A checker that flags half your files does not get tightened. It gets muted. The cost of a false positive is not one wasted look, it is the credibility of every finding after it, including the true ones sitting three lines down.
I went from 50% to 3% by making the tool refuse to guess: no bare word is a command, no slash is a file unless it starts somewhere real, no relative path gets resolved against a directory the checker invented, no date is a deadline without a word saying so. Every one of those was me deciding that reporting nothing beats reporting something shaped like an answer.
And the last false positive did not fall to a better regex. It fell because I checked a finding by hand and found my own script wrong. If you build one of these, budget for that step. The auditor is a claim about your claims, and it decays the same way.
The script is 260 lines of standard library Python, no dependencies. Point it at a memory directory:
memrot.py ~/.claude/projects/<project>/memory --op
The --op flag is 1Password specific and shells out read only. Drop it and the rest still runs.
Some filenames in the output above are edited. The memory holds personal records, and the structure was the point, not the contents.
Top comments (10)
This mirrors a setup I run almost exactly — one fact per file, frontmatter, an index that loads at session start — so the false-positive triage rang very true.
The backticks-are-not-commands trap is the one that bites hardest, because agent-written memory uses backticks as emphasis, not as shell syntax. Your "name plus a real argument or a known CLI" heuristic is the right call, and I'd defend the allowlist harder than you did: for a nagging tool, precision beats recall every time. A checker that cries wolf on
robustgets muted within a day, and then it catches nothing. Better to miss a few real rots than to train yourself to ignore it.The highest-value checkable in my experience is the shelf-life date — an explicit "revalidate after" beats trying to infer staleness from the claim's content. And I'd resist auto-repair: flag-and-surface keeps a human (or the agent, deliberately) in the loop, because a confidently-wrong "fix" to a memory file is worse than a stale fact you know is stale. Did you find grading staleness (fresh / suspect / dead) more useful than the binary, or did the three-state version just add noise?
No three-state version. I split by kind of claim, not by confidence: an expired shelf life left the rot count entirely, because stopped being actionable and was never true want different reactions from me.
Shelf life is the best checkable, agreed. Reran it today: 6 dated claims past their horizon, none of them rot.
On auto-repair the number is on your side. The first run flagged 45 files and 42 of those were the checker being wrong. Write access would have deleted 42 true facts and left me the three real ones.
This is the first memory-auditor writeup I’ve seen where the auditor itself gets audited. “A relative path is a claim about a context the checker doesn’t have” is the bit I’m stealing.
That line started as a bug report against myself. The auditor was resolving
./scripts/gate.shagainst its own cwd and calling a file that exists dead.Same shape got me on the 1Password check: the script asked
op item get, and the id turned out to name a vault. Nothing rotten in the memory, the checker just asked the wrong question and reported the answer as a fact.Steal away.
The strongest part of this is that you audited the auditor instead of trusting the first alarming number.
There is one question I think remains open after going from 45 findings to 3:
How many genuinely stale facts did the stricter verifier stop detecting?
The manual review gives you useful evidence about false positives and precision. It does not yet measure false negatives or recall.
A small negative-control test could close that gap:
That would separate two claims:
Your article provides good evidence for the first. The controlled corpus would test the second.
I’m developing an independent verification method called System Claim Check for bounded system claims like this. I have a few free pilot checks available. If you are interested, I’d be happy to help define and independently evaluate that controlled test, then return a documented verdict publicly or privately.
You are right, and I did not measure it. The article is evidence on precision, none on recall.
The gap splits in two. Recall lost by tightening is bounded: the 41 findings the loose run made and the strict run dropped, so diffing the two runs measures it without planting anything. Recall the tool never had is the other half, and a planted corpus only reaches inside the five categories it already claims. The claims that matter most are the ones no script can check.
I will run the diff and post the numbers, the worse ones included.
Numbers, as promised. Same corpus of 103 files, both versions of the checker.
Loose: 71 files, 327 findings. Strict: 10 files, 13. So 314 dropped, nothing added.
I read all 314: Homebrew casks, Python packages, CI tools, blog tags, API routes, and a vault id holding six live items.
Tightening cost no recall I can measure.
The op read vs op item get split is the part I felt. Same id, one call resolves archived items, the other refuses and hands you "deleted or archived" weighted equally. I got burned by that exact phrasing once and now treat any either-or error message as a lookup bug until proven otherwise, because whoever reads it, agent or human, will pick the cheaper interpretation.
The false positive math also matches what I have seen. A checker reporting 50% rot does not get tightened, it gets muted, and the real "gone from 1Password" line three rows down dies with it. We had the same dynamic with a flaky contract test suite: fixing signal-to-noise did more for safety than adding any new check.
Do you run the auditor on a schedule, or only on demand? Curious whether the 3% drifts as the memory gets older.
On demand, no schedule. Your question caused one, so: 103 files now against the 90 in the post, three days later.
ROTTEN: 7/103 (6%)looks like drift until you check it. Four of the six dead paths sit on an external SSD that is not plugged in. The missing command isruff format, a fact about CI and not about this laptop. Two are real.So rot held at two files, and both new findings were new false positive classes. The ceiling is the checker, not the memory.
This is a really interesting example of why memory validation can't just be a “does this fact still exist?” check.
What stood out to me most was the distinction between a false statement and a statement that is no longer actionable. The expired date example makes that especially clear.
I also like the final lesson: the auditor itself has to be treated as something that can be wrong. Otherwise you end up building a second layer of confident misinformation on top of the first.
It makes me wonder whether the next step for project memory is validating not just individual facts, but the dependencies and assumptions behind them. A path can still exist while the architectural decision that depended on it is no longer valid.
Curious if you see that as a natural extension of memrot, or intentionally outside its scope?