57% of organizations now have AI agents running in production, according to a 2026 Gartner survey. Here's what almost none of them will tell you: the memory system is probably lying to the agent.
Not maliciously. Just... confidently. Because every piece of stored memory carries the same authority, and that's a structural problem you don't notice until you've been burned by it.
I've been running multi-agent systems in production since the start of the year. This is what I learned.
The Three-Tier Model Nobody Fully Explains
When I started, I treated "agent memory" as a single thing — vector store, retrieval, done. After six months, I think about it as three distinct systems that happen to live in the same pipeline:
Episodic memory — what happened in specific interactions. Conversation history, action logs, task outcomes.
Semantic memory — facts about the world that the agent has learned and should retain. User preferences, project context, anything that generalizes across sessions.
Procedural memory — how to do things. Agentic workflows, tool chains, the agent's own operating procedures.
The reason this split matters: they fail differently.
Episodic memory fills up fast and needs aggressive summarization or you hit context limits. Semantic memory has a contradiction problem — old facts and new facts fight each other and the agent has no way to weight them. Procedural memory is the most stable but the hardest to update without breaking existing workflows.
Most articles treat these as one problem. They're not.
The Architecture That Actually Works (With Code)
After going through Mem0, Letta, and a few custom implementations, I settled on a tiered architecture that looks like this:
class AgentMemorySystem:
def __init__(self):
# Tier 1: Episodic — short-term, high-fidelity
self.episodic = VectorStore(
collection="episodes",
embedder="text-embedding-3-small"
)
# Tier 2: Semantic — long-term, summarised
self.semantic = VectorStore(
collection="facts",
embedder="text-embedding-3-small"
)
# Tier 3: Procedural — stable, versioned
self.procedural = DocumentStore(
collection="procedures",
versioned=True
)
def store_episode(self, interaction: dict):
summary = self._summarise(interaction)
# Store last 50 episodes raw, older ones as summary
if self.episodic.count() < 50:
self.episodic.add(interaction)
else:
self.episodic.add(summary)
def store_fact(self, fact: dict, confidence: float = 1.0):
# Attach confidence metadata — this is the authority signal
fact["confidence"] = confidence
fact["timestamp"] = datetime.utcnow().isoformat()
self.semantic.add(fact)
def retrieve(self, query: str, memory_type: str = "all") -> list:
if memory_type == "all":
return (
self.episodic.search(query, top_k=5) +
self.semantic.search(query, top_k=3) +
self.procedural.search(query, top_k=2)
)
return getattr(self, memory_type).search(query, top_k=5)
The key addition here is the confidence field on semantic facts. This is the piece that fixes the "equal authority" problem. New information from a trusted source (the user, a verified tool) gets confidence=1.0. Old information that was inferred rather than stated gets confidence=0.7. The retrieval pass weights by confidence before passing context to the LLM.
The Failure Modes Nobody Warns You About
1. The Summarisation Spiral
Every week, episodic memory needs to be summarised or you start dropping context. But summarisation is lossy. Run it too aggressively and you lose the specific details that made the original interaction useful.
My rule: summarise at 50 episodes, but keep the last 10 raw. The most recent interactions are the highest signal.
2. The Contradiction Cascade
User told you their name was "Alex" in January. In March they said "I go by Alexei now." Both facts live in semantic memory. The agent retrieves both and now you have a conflict that it will surface at the worst possible moment.
The fix: semantic updates should soft-delete the old fact rather than overwrite it.
def update_fact(self, entity: str, new_value: dict):
# Soft-delete old facts for this entity
self.semantic.soft_delete(f"entity:{entity}")
# Store new fact with higher confidence
new_value["confidence"] = 1.0
self.semantic.add(new_value)
3. The Procedural Drift
Procedures are supposed to be stable, but tool versions change. An agent that learned to use the Slack API in January is still running the same procedure in August. It's not wrong, but it's not optimal either.
Procedural memory needs a TTL and a re-onboarding mechanism. I add last_validated timestamps to all procedures and run a quarterly review pass.
How to Evaluate Whether Your Memory Is Working
Here's the test I run every sprint: I take the last 20 agent actions and manually check whether the agent's "reasoning" about what it remembered was accurate.
Specifically:
- Did it retrieve the right memory?
- Did it interpret the confidence correctly?
- Did the retrieved context actually change the output, or was it decorative?
If retrieval changed the output less than 30% of the time, your memory system is decorative. It's storing things but not using them effectively.
For the more formal route, LoCoMo is the standard benchmark (Mem0 hit 92.5 on it). But I find the sprint test more useful because it tells you whether the memory is actually helping the agent in your specific use case, not just in a synthetic evaluation.
What I Learned
The biggest shift for me was moving from "agent memory as a feature" to "agent memory as infrastructure." You don't bolt it on after the agent is working. You design it from the start, with explicit decisions about:
- What gets remembered and what gets discarded
- How much authority each memory tier carries
- What happens when memories contradict each other
The agents that feel the most capable aren't the ones with the biggest context windows. They're the ones with the best memory hygiene.
Running multi-agent systems in production since 2026. Writing about what actually works.
Top comments (0)