My AI agents read a written memory before they do anything. Notes about how a script works, what a decision was, which file owns what. It works well, and it has one failure mode that matters more than the rest.
The notes go stale. Quietly.
A stale note is worse than a missing one. A missing note makes the agent go look. A stale note makes it confident and wrong.
So I set out to have the machine catch it for me. I wrote three heuristics. All three failed, and the way they failed is the interesting part.
The obvious one, which is sound, and found nothing
Here is the check anyone would write first. If a note quotes something specific about a file, a --flag or a functionName(), go read that file. Is the thing still there? If it is gone, the note is stale.
It is sound. It is cheap. It has almost no false positives by construction, because it only fires on an exact string that used to exist and now does not.
I ran it across fifty real note-and-source pairs.
Zero hits.
My first instinct was that I had a bug. I did not. The check was working perfectly and there was simply nothing for it to find.
Why the null result is the actual finding
Sitting with that zero taught me more than a list of hits would have.
The staleness that actually bites is not subtractive. It is additive.
The problem is almost never "this note says something that is now false." It is "this file grew something the note never mentions."
One of my scripts quietly gained a whole new subcommand and a new ownership flag. The note describing that script was not wrong about anything. Every word in it was still true. It just listed four commands when there were now six. An agent reading it would never learn the new ones existed, and would never have a reason to doubt what it read.
That is the shape of every real case I found. Nothing contradicted. Something omitted.
And you cannot write a deterministic check for it. Look at a diff of added lines and ask "did some note owe this a mention?" That is a semantic question about intent. The file gained a function. Does the note about it need to say so? Sometimes yes, usually no. There is no rule that separates them.
Which is why the boring answer wins. A human stamps a date on the note saying "I confirmed this." The machine compares that date against when the file last changed. It never has to understand anything. It converts an unanswerable question into a date comparison and gets the answer from the only thing that can actually judge it.
I did not want that answer. I wanted the clever one. The clever one does not exist.
The other two, because they are what you would try next
Mention count. If lots of notes reference a file, that file is important, so rank it higher for review. Reasonable. It ranked my one false positive above my one true positive. Popularity is not staleness, and it turns out the most-mentioned files are the ones people keep updating, which makes them the freshest things in the system.
Fan-in. Same idea from the other direction: rank a note by how many other notes point at it. This scored my central registry, the single most load-bearing file I have, below noise. It is pointed at constantly and it changes constantly, and the metric could not see either.
Both are proxies. Both measure attention and call it decay. They are not the same thing and they are sometimes opposites.
What I actually shipped
A stamp, and a checker that compares two dates. Thirty-six tests. It found forty-eight things worth looking at and I drained them to zero the same day.
Two of them were real staleness of the additive kind, and neither would have been caught by anything I had built before.
The part I keep coming back to
I broke this tool three times while draining it. Every single break made it report less than the truth.
One was a date filter that silently stopped searching early. One wrote a stamp it could not read back, and reported success on fifteen files that every other check still saw as unstamped. One anchored to midnight, so it went quiet on exactly the files I was working on that day.
Zero findings and a healthy tool look identical from the outside. So the checker now refuses to report zero without also proving it actually looked at something.
Three failed heuristics taught me more than the working one. The measured failure tells you the shape of the problem. The confident guess just tells you what you already believed.
Top comments (2)
This is the failure mode I keep coming back to with agent memory. Contradiction is easy to test for, but missing coverage is where the cost hides. The practical check is probably not whether the note is true. It is whether the note still predicts the interface someone will actually use.
That's a better way to say it than I managed in the post. Whether the note still predicts the interface someone will reach for is closer to what I was trying to get at.
The case that made me give up on the clever version fits it exactly. A note listed four commands and the script had six. Nothing in it was false. It just described a smaller interface than the one that existed.
I still don't know how to test that without a human in the loop. Deciding which additions owed a mention is the same judgment call, moved one step later, which is how I ended up at a date stamp instead. If you've got something that works better, I'd genuinely like to see it.