In August my coding agent kept telling me that one kind of article on this site draws about four times as many comments as another. A week earlier I had corrected that number myself: with more data it was closer to two to one.
The old note was still in its memory. It sounded exactly as sure as the new one, and it kept winning, because nothing about it looked old.
That's the failure I find hardest to catch. A missing memory is loud: the agent asks, you answer, done. A stale one is quiet. It hands you a confident answer that stopped being true, and it reads like every correct answer it has ever given you.
I tried to fix it by ranking older notes lower, and that went badly.
So I'm collecting real cases for the next part of this series:
- What did your agent remember that had stopped being true? A config value, a renamed API, a decision your team reversed, a workaround for a bug that's long fixed.
- How did you notice? A failing build, a reviewer, a customer, luck?
- How long was it wrong before anyone saw it?
Short answers are welcome. The ones with a concrete example are the ones I'll learn the most from.
Top comments (7)
During release checks one of my agents read production at about 12:50 and found the default gate not set, then kept repeating "not set" all afternoon without checking again. I only noticed because I knew the gate was set and told it so. I still don't know whether the setting changed after 12:50 or it misread it the first time.
The part I'd find hardest is the one you can't answer: whether the gate changed after 12:50 or was misread. The agent turned a reading into a fact and dropped the time it was taken. "Not set as of 12:50" would have told you exactly what to re-check.
One of our own CI checks had the same shape. It read a pull request's title once, when the run started, and kept reporting "no card in the title" after someone had fixed it. Re-running didn't help, because the re-run replayed the same snapshot. What fixed it was reading the live state every time instead of trusting the stored one.
Would you mind if I used your release-gate case in the next part of the series? I'd describe it the way you did here.
I build Aladdin, a job agent. The stale case I watch for is a form answer a user corrected once, then the agent keeps reusing the old version because it sounds confident.
My fix is boring on purpose. A fresh worker reads the actual page and form for each application, and it tailors only from the user's own facts. If the form cannot be read or an answer is unknown, it asks the user instead of guessing.
That does not remove stale facts entirely. It just makes the failure loud again, which is the part I want.
A fresh worker that reads the actual form makes the page the source of truth, and memory only supplies the user's own facts. That moves the stale-answer risk into one place: the user's facts themselves.
That's where the correction case still bites for me. After a user corrects something, the old fact and the new one look equally confident. In my store, a newer entry can name the one it replaces, so the old one stays readable but stops winning. The catch is that agents almost never write that link on their own: when I counted, 2 of 735 entries carried one.
Stale memory is often more dangerous than missing memory because the agent has no obvious reason to question something it previously learned. I'd add one dimension to memory management: every fact that can change should carry its source, last-verified timestamp, and validity conditions. A corrected value shouldn't simply compete with the old one on relevance or ranking; it should explicitly supersede it.
At IT Path Solutions, this is an important consideration when designing production RAG and agentic systems. I'd also distinguish between facts, temporary observations, and decisions. A workaround that was valid for one software version shouldn't become permanent guidance, and a team decision that gets reversed needs to invalidate the earlier instruction.
The real test isn't whether an agent can retrieve something it remembers. It's whether it can recognize when that information is no longer safe to rely on.
This matches what I have seen too. A missing fact is easy to spot because the agent asks. A stale fact is quieter, because the agent sounds confident while it repeats something I already corrected. My rule now is that a correction should replace the old note, not sit next to it, and the current source should win when they disagree. Thanks for asking for concrete cases, the release gate example above is a good one.
Had this happen with a Cloudflare D1 binding ID and a local dev port I had changed in wrangler.toml. Even after I corrected it, the agent kept running migration scripts against the old local DB ID for about 25 minutesโthe migrations succeeded cleanly against the dead instance while my app kept throwing missing-table errors. Now I just force it to read the live config file before running any CLI command instead of trusting its session notes.