Most agent failures aren't model failures. They're memory failures.
- Bad encoding
- Noisy storage
- Chaotic retrieval
- Misaligned pruning
If you've watched an agent confidently retrieve last year's policy, or hallucinate because its context window filled with garbage, you've seen memory drift in the wild.
This post gives you a structural model and code patterns to make memory architecture a first-class engineering object.
The Two Loops
Inner Loop = runtime behavior
Outer Loop = architecture evolution
Most frameworks only implement the inner loop. That's why drift accumulates silently.
class Agent:
def inner_loop(self, task):
encoded = self.memory.encode(task)
self.memory.store(encoded)
context = self.memory.retrieve(task)
output = self.model.run(task, context)
self.memory.manage(task, output)
return output
def outer_loop(self, logs):
diagnostics = analyze(logs)
self.memory.redesign(diagnostics)
The inner loop learns. The outer loop redesigns.
If you don't have both, you're shipping a student who never upgrades their study method.
The Four Rooms
Every memory system has four components. When something breaks, debug the room—not the agent.
class Memory:
def encode(self, item):
return embed(item) # embedding model, chunking, feature extraction
def store(self, vector):
vector_db.insert(vector) # vector DB, KV store, graph
def retrieve(self, query):
return vector_db.search(query, top_k=5) # similarity search, reranking
def manage(self, task, output):
prune_stale()
reindex()
decay()
| Room | Drift Pattern | Symptom |
|---|---|---|
| Encode | Embeddings lose contrast | Everything looks similar |
| Store | DB becomes a hoarder's attic | Bloat, slow queries |
| Retrieve | Top-k returns stale/irrelevant items | Wrong context, hallucinations |
| Manage | Pruning removes wrong things | Lost knowledge, unstable behavior |
Drift Detector
def detect_drift(memory):
return {
"encoding_variance": variance(memory.embedding_stats),
"storage_growth": memory.db.size(),
"retrieval_accuracy": memory.metrics.retrieval_precision(),
"pruning_errors": memory.metrics.prune_misses()
}
If retrieval accuracy drops while storage growth spikes, you're in classic slop territory.
Governance Toolkit
Governance isn't compliance. It's maintenance.
# === APPRENTICE LOOP (Weekly) ===
# Surface friction from runtime behavior
def apprentice_loop(agent, tasks):
return [(task, agent.inner_loop(task)) for task in tasks]
# === ARCHITECT LOOP (Monthly) ===
# Redesign the structure that produced the friction
def architect_loop(agent, logs):
agent.memory.redesign(analyze(logs))
# === FOUR ROOMS AUDIT (On Drift) ===
# Diagnose which room failed
def audit(memory):
return {
"encode": memory.encode_stats(),
"store": memory.db.health(),
"retrieve": memory.metrics.retrieval_precision(),
"manage": memory.metrics.prune_misses()
}
# === DRIFT WATCH (Continuous) ===
# Catch slop early
def drift_watch(memory):
if memory.db.size() > MAX_SIZE:
warn("Storage overgrowth")
if memory.metrics.retrieval_precision() < THRESHOLD:
warn("Retrieval drift")
if memory.embedding_stats.variance < MIN_VARIANCE:
warn("Encoding drift")
# === ARCHITECTURE LEDGER (Versioning) ===
# Track how memory evolves
def log_change(change):
with open("architecture_ledger.jsonl", "a") as f:
f.write(json.dumps(change) + "\n")
If you don't version your memory architecture, you're one schema change away from chaos.
The Point
As agents become more autonomous, the memory system becomes the real engine. Not the model. Not the prompt. Not the RAG pipeline.
The architecture is the behavior.
If you want predictable agents, you need predictable memory.
If you want predictable memory, you need governance.
If you want governance, you need the two loops and the four rooms.
For the conceptual framework behind this post, see The Two Loops on Substack.
Top comments (0)