The problem in one paragraph
An agent's memory system recorded your deploy target in March. In July the target changed. The system still retrieves the March record with full confidence, and the agent acts on it. Nothing failed in the retrieval pipeline; the stored belief simply stopped being true. That gap, between a fact changing in the world and a system's stored state catching up, is state-drift.
Not data drift
State-drift is not data drift or concept drift. Those terms describe shifts in a model's input distribution, the thing observability tooling detects and alerts on. State-drift is about stored beliefs going stale: the inputs can be perfectly in-distribution while the memory that interprets them is out of date.
What the field has measured
STALE (arXiv:2605.06527, May 2026) built 400 expert-validated conflict scenarios and asked whether LLM agents can tell when their memories are no longer valid. The strongest frontier model reached 55.2% overall accuracy. Most specialized memory frameworks fell below 10%.^(1)1 Every figure in this paragraph is the STALE authors' own, as reported in their paper. The lab produced none of it. The paper's core observation is a gap between retrieving updated evidence and acting on it: retrieval is not the bottleneck, acting on what was retrieved is.
ForgetEval (arXiv:2606.15903, June 2026) studied forgetting across thirteen agent-memory configurations. Two results stand out. First, production failures are predominantly forgetting failures: rotated credentials still recommended, deleted records still surfacing. Second, when leading memory systems were scored on test cases written by someone other than their own authors, in-house scores of 63 to 68 percent dropped to 28 to 33 percent.^(2)2 The comparison is between test cases written by a system's own authors and test cases written independently. The drop from 63 to 68 percent to 28 to 33 percent is the ForgetEval authors' result. The drop is the point: who writes the test cases matters.
Why it's a security problem
A stale belief is a reliability bug when the world changes on its own, and a vulnerability when someone changes it on purpose. The mechanism is the same. If a system can be made to hold a fact past its expiry, a rotated credential, a revoked permission, a decommissioned host, a superseded instruction, then whoever controls the timing of the update has partial control over what the agent does next.
Read ForgetEval's production failures as a threat model rather than a bug list and the point lands immediately: rotated credentials still recommended, deleted records still surfacing. The lab's benchmark was built to measure how shipped systems fail on their own, and is archived. The red team work that looks for the cases where they can be made to fail is now part of Stonepath IT Research Labs, and what it finds informs the defensive work on this site.
What remains untested
The recall plane, retrieving stored facts, is extensively benchmarked. The control plane, the operations that supersede, release, and purge those facts, has far less coverage against production systems as they actually ship.^(3)3 The recall plane is retrieving stored facts. The control plane is the operations that supersede, release, and purge them. The library entry for ForgetEval says what the study covers and where it stops. Several efforts measure aspects of belief updating; the library keeps an annotated account of what each covers and where it stops.
The WasTrue Benchmark
The lab built a benchmark to run production memory systems as deployed, under changing facts, in an operational harness: Mem0, Zep, Letta, Anthropic's memory tooling, and dreamd. The dataset, oracle, and fairness scripts were to publish alongside the results, so anyone could re-run the comparison or write their own probes.
It was archived in August 2026 before any of that shipped. No results were produced for publication and none are claimed here; every figure on this page belongs to someone else, cited above.^(4)4 The findings record lists the harness and its results as archived and never published.
Notes
- Every figure in this paragraph is the STALE authors' own, as reported in their paper. The lab produced none of it. ↩
- The comparison is between test cases written by a system's own authors and test cases written independently. The drop from 63 to 68 percent to 28 to 33 percent is the ForgetEval authors' result. ↩
- The recall plane is retrieving stored facts. The control plane is the operations that supersede, release, and purge them. The library entry for ForgetEval says what the study covers and where it stops. ↩
- The findings record lists the harness and its results as archived and never published. ↩
References
- Chao, Bai, Sheng, Li, and Sun (May 2026). STALE: Can LLM Agents Know When Their Memories Are No Longer Valid? arXiv:2605.06527. arxiv.org/abs/2605.06527
- Control-Plane Placement Shapes Forgetting, introducing ForgetEval (June 2026). arXiv:2606.15903. arxiv.org/abs/2606.15903
- The lab's annotated account of both, with what each covers and where it stops. botzr.com/library
Originally published at botzr.com.
Top comments (0)