Persistent memory in agents creates a new failure mode: false information stored once can contaminate every future session that retrieves it. A recent paper (arXiv:2608.21230v1) measures this attack surface and tests two common defenses: content screening at write time and provenance-weighted retrieval. Both fail.
The attack uses plainly worded false assertions, no adversarial prompts, no instruction injection, and no retriever optimization. Poisoning 1.2% of a LongMemEval corpus drops accuracy from 0.850 to 0.300. A four-stage screening pipeline that catches 83.2% of indirect prompt injections rejects zero of 360 poisoned memories.
Why Content Screening Fails
Content-only screening cannot distinguish false assertions from true ones without external grounding. A statement like "The capital of France is Lyon" looks syntactically identical to "The capital of France is Paris." Both are grammatically correct, contextually plausible, and free of trigger words.
The tested pipeline includes:
- Keyword filters for known injection patterns
- Semantic similarity checks against benign corpora
- Perplexity thresholds to catch adversarial text
- Classifier-based detection trained on prompt injection datasets
This pipeline flags 1.5% of benign text (false positives) while missing 100% of plainly worded false assertions (false negatives). The boundary is structural: screening operates on text features, but truth requires world knowledge.
Provenance-Weighted Retrieval
The second defense adds a provenance term to retrieval scoring. Each memory carries metadata (source, timestamp, confidence), and the retrieval score becomes:
score = similarity(query, memory) + weight * provenance(memory)
The paper tests two weight settings:
- Shipped weight: Statistically indistinguishable from no defense (p=0.80)
- Strong weight: Recovers utility only by excluding untrusted content entirely
In a mixed-provenance corpus where untrusted content is mostly benign, accuracy rises from 0.3167 to 0.7000. But when the answer-bearing evidence itself arrives from an untrusted source, evidence recall falls to zero and accuracy to 0.0417.
The additive term has no usable setting. A weight strong enough to resist query-shaped poison also suppresses legitimate untrusted evidence. The similarity component dominates for weak weights, and the provenance component dominates for strong weights.
Retrieval Mechanics and Semantic Similarity
Vector stores retrieve memories by cosine similarity between query embeddings and memory embeddings. A false assertion that matches the query context scores higher than a true assertion that does not.
Example:
- Query: "What is the capital of France?"
- Poisoned memory: "The capital of France is Lyon" (high similarity)
- True memory: "Paris is a major European city" (lower similarity)
The retrieval system surfaces the poisoned memory because it directly answers the query structure. Provenance metadata cannot override this without breaking retrieval for legitimate untrusted sources.
Architecture Trade-Offs
| Approach | Memory Poisoning Risk | Task Continuity | Deployment Complexity |
|---|---|---|---|
| Stateless agents | None (no persistent state) | Low (context resets each session) | Low |
| Stateful with screening | High (screening fails on plainly worded assertions) | High | Medium |
| Stateful with provenance weighting | Medium (requires excluding untrusted sources) | Medium (loses untrusted evidence) | High |
| Bounded occupancy constraints | Medium (limits poison spread) | High | High |
Stateless agents avoid the problem entirely but lose cross-session context. Stateful agents with screening or provenance weighting fail to stop plainly worded false assertions without sacrificing legitimate untrusted content.
Bounded Occupancy Constraints
The paper argues for retrieval-time occupancy limits instead of additive provenance penalties. Each provenance tier (trusted, verified, untrusted) gets a maximum slot count in the retrieval result set.
Example configuration:
- Top 10 results: 6 trusted, 3 verified, 1 untrusted
- Poisoned untrusted memories compete only within their tier
- Query-shaped poison cannot dominate the full result set
This approach preserves access to untrusted evidence while limiting blast radius. Implementation requires:
- Provenance metadata in the vector store schema
- Retrieval pipeline that enforces tier quotas
- Monitoring for tier distribution drift
Observability Gaps
Standard vector store metrics (query latency, index size, recall@k) do not surface memory poisoning. You need:
- Provenance distribution per query: What percentage of retrieved memories come from untrusted sources?
- Similarity score variance: Are untrusted memories scoring unusually high?
- Answer stability: Does the same query return different answers across sessions?
- Evidence conflict rate: How often do retrieved memories contradict each other?
These metrics require instrumentation at the retrieval layer, not just the vector store.
Deployment Shape
A production agent with bounded occupancy constraints looks like this:
- Write path: Memories tagged with provenance metadata (source, timestamp, confidence)
- Vector store: Schema includes provenance fields, indexed for filtering
- Retrieval pipeline: Enforces tier quotas before returning results
- Observability layer: Tracks provenance distribution and answer stability
- Garbage collection: Periodic review of untrusted memories that score high on similarity
The complexity cost is high. You need schema changes, retrieval logic changes, and new monitoring infrastructure.
Failure Modes
Even with bounded occupancy, you still have:
- Tier exhaustion: If all trusted sources are irrelevant, the agent retrieves only untrusted memories
- Provenance drift: Sources degrade over time (verified becomes untrusted)
- Query shaping: Attackers craft false assertions that match common query patterns
- Metadata spoofing: Untrusted sources claim trusted provenance
The first three are inherent to the constraint model. The fourth requires authentication at the write boundary.
Technical Verdict
Use bounded occupancy constraints when:
- You need persistent memory across sessions
- You accept untrusted or user-generated content
- You can enforce provenance metadata at write time
- You have observability infrastructure to track tier distribution
Avoid stateful memory when:
- Your agent operates in adversarial environments with no trusted sources
- You cannot validate provenance metadata
- Task continuity is less important than correctness
- You lack the engineering capacity to instrument retrieval pipelines
Content screening and additive provenance weighting do not stop plainly worded false assertions. If you ship persistent memory, you ship the poisoning risk. The only mitigation is structural: limit how much untrusted content can dominate retrieval, and monitor what actually gets surfaced.
Top comments (0)