Cross-post of Insights #17. Canonical: https://sheikhwasim.com/insights/memory-expiry-agent-lessons-need-a-source/
The agent remembered a lesson from a bad Tuesday. It kept applying it for four months.
Persistent memory is the feature everyone wants in their agents right now. Let the agent learn from past runs, save what worked, and stop repeating mistakes. It sounds like pure upside. In production, it has one quiet failure: a lesson saved for a good reason on one day keeps getting applied long after the reason is gone, and nobody can tell where it came from.
The fix is Memory Expiry: every saved lesson carries its source, its scope, and a date when it must be rechecked. A lesson nobody can trace back does not get used.
Quick use case
Situation. A payments team runs a coding agent that opens pull requests for small fixes: dependency bumps, lint cleanups, minor bugs. The agent has long-term memory. After each run it can save short lessons it will read at the start of future runs.
What broke. During one incident, the shared test sandbox was down for most of a day. Integration tests timed out on every run. The agent, trying to finish its task, saved a lesson: "Integration tests on the payments repo are flaky. Rely on unit tests." It was a reasonable note that day.
The sandbox came back that evening. The lesson stayed. For four months, the agent skipped integration tests on every payments pull request, and its PR descriptions said "unit tests pass," which was true. Reviewers saw green checks and approved. Then a currency-rounding change shipped that unit tests could not catch, and a batch of refunds went out a few cents wrong.
When the team went looking for why the agent skipped those tests, they found the lesson in memory with no date, no link to the run that created it, and no note about the outage. Nobody could say whether it was ever true.
The fix (Memory Expiry).
- Source on every lesson. Each saved memory stores the run ID it came from and the evidence that triggered it, for example "sandbox timeout, 14 of 14 integration tests, Jul 8."
- Scope on every lesson. The lesson applies to one repo, one task type, or one tool, never "everything."
- Expiry on every lesson. Lessons get a recheck date. Anything that tells the agent to skip a safety step gets the shortest one, a few days.
- Recheck, don't renew. On expiry, the agent retests the claim (run the integration tests once). If the claim no longer holds, the lesson is deleted, not extended.
- No source, no use. Lessons without a traceable run go to a review queue instead of into the prompt.
Same agent, same memory store. After the change, the "flaky tests" lesson would have expired in three days, the recheck would have passed, and the note would have been gone before the rounding change existed.
Why agent memory goes stale
A model's knowledge is frozen at training time, and everyone plans for that. Agent memory looks different because it is written during real work, so it feels current. But each lesson is a snapshot of one moment: one outage, one flaky API, one reviewer's preference, one workaround for a bug that has since been fixed.
Memory also compounds. A lesson read at the start of every run shapes every run, and the agent rarely questions its own notes. A wrong memory is worse than no memory, because it carries the confidence of experience.
Memory Expiry (four layers)
1. Write with provenance
The memory write path requires a source run, the evidence, and the time. If the agent cannot point to the run, the write is rejected. This is cheap to enforce in code and impossible to rebuild later.
2. Retrieve by scope
A lesson about the payments repo is not loaded for the billing repo. Retrieval filters on repo, task type, and tool before anything reaches the prompt. Narrow scope keeps one bad day from spreading.
3. Expire by risk
Set expiry by what the lesson does, not by how old it is. Style preferences can last months. Anything that skips a check, widens access, or changes what gets tested expires in days. Same thinking as the Permission Envelope: trust that lasts forever was never really granted.
4. Recheck on expiry and on incidents
When a lesson expires, the agent tests the claim before keeping it. When an incident happens, list every memory loaded in the runs involved. Those lessons are suspects until rechecked. Store the result in the trace, next to the Golden Trace you already compare against.
How this maps to what you already have
- 5-layer agent stack. Memory sits in State, and its writes and reads should show up in Traces.
- Done Check. A Done Check proves a task is right today. Memory Expiry makes sure yesterday's shortcut doesn't weaken tomorrow's check.
- Blast Radius. Scope limits on memory are a blast radius for lessons.
- Ship Gate. Add one line: no agent with long-term memory ships unless every memory write has a source, a scope, and an expiry.
A test you can run this week
Export your agent's memory store, or whatever it reads at the start of each run. Pick 20 entries.
Ask three questions for each one: (1) Can you find the run that created it? (2) Is it limited to the place it was learned? (3) Would it still be true if you checked today?
If more than a couple fail the first question, stop writing new memories until the write path records a source. Then delete or quarantine anything that tells the agent to skip a step and can't be traced.
Closing
Agents that learn are worth building. Agents that never forget are a liability. Give every lesson a source, a scope, and a recheck date, and treat untraceable memory as a guess rather than experience. The goal isn't an agent with more memory. It's an agent whose memory you can audit.
I'm Wasim Sheikh, AI Architect. I build systems teams trust and organizations depend on: not demos, not proofs of concept, production.
Follow for practical AI architecture that ships.
Connect on LinkedIn: Wasim Sheikh · Site: sheikhwasim.com · Notes: Wasim Sheikh on Substack (practicalainotes.substack.com) · X: @anciwasim
Top comments (0)