We've spent two years making models smarter and context windows bigger. But here's the dirty secret of production agents: most of them "remember" by overwriting a variable. That's not memory. That's a bug wearing a trench coat.
Your agent remembered the user "doesn't like X" yesterday. Today it hands them a plan built on X. You check the log and find one line: pref.x = like. Who changed it? When? On what basis? Nobody knows. Not you, not the agent.
This isn't a model failure. It's an architecture failure — and it's the single most common one in agent memory today.
The core idea in one sentence
Most agent "long-term memory" fails not because software didn't store data, but because memory was wrongly modeled as an overwritable state variable. The correct architecture is memory = an append-only event log (WORM) + a materialized view projected on demand. The write side is forever traceable and never overwrites; the read side forever projects "the current single source of truth."
"Being able to store" and "being able to store trustworthily" are two different things. A paid decision-maker buys the second. An overwriteable state store is engineering's default laziness; append-only + materialized view is the step that turns memory from "data" into "evidence."
Why this matters now
Three structural shifts are happening in production, not in theory:
- Agents went from single-turn assistants to autonomous systems that rewrite their own config. The moment the writer of memory stops being "one trusted, reviewed programmer" and becomes "a possibly-concurrent, possibly-wrong, non-deterministic model," writing stops being an operation and becomes a source of incidents.
- Tool-call protocols (MCP-style) give the agent a peripheral bus — but memory sits outside that bus. MCP solved "how does the agent call tools," not "how does context survive reliably across sessions." Tools are legs; memory is the brain.
- Evidence / audit / rollback went from a compliance checkbox to a debugging necessity. A production agent corrupts its own memory at 3 a.m. and breaks everything downstream. The next morning you don't need "it's broken now" — you need the replay: which steps, what did it write, why.
The fix: append-only log (WORM) + materialized view (MV)
Two iron rules decide everything:
-
Write side: only append, never update. Every fact / preference / conclusion lands as an immutable event carrying timestamp, actor, and rationale. Even a correction is a new
retract/supersedeevent — express invalidation by appending, never by overwriting history away. - Read side: only read from the materialized view, never do mutable aggregation on the evidence layer. The projector folds the log into a "current state snapshot" (latest-wins, or business-rule merge). The snapshot can be cached, overwritten, even lost and rebuilt from the log — the view is expendable; the log is not.
Your 30-minute architecture health-check
-
Is your memory layer a "state store" or "log + view"? Open the schema; any
UPDATE/UPSERT/overwriteon a memory entity ⇒ you carry the three risks above. AllAPPEND/INSERT+ reads via projection ⇒ you pass. -
Turn correction into an appended
retractevent. Addsupersede {target_event_id, new_value, actor, rationale}. Test: any row must answer which supersede/retract still reference it. -
Force three required metadata on every write:
ts,actor,rationale. Test: no new write point may skip these three. -
Make the read-side "current state" a swappable projection:
current_snapshot := project(event_log, merge_strategy). Start withlatest_wins. - Split evidence vs view consistency: log = append-only, permanent, strongly consistent; view = rebuildable, disposable, eventually consistent.
- Add a guardrail on autonomous edits: before an agent appends, require "rationale must have a source". Else route to a "pending adjudication" queue.
Honest objections
- "Human memory is overwrite/forgettable; why not copy that?" Human forgetting is a passive failure to separate evidence from noise — not a feature. Copy the auditable replay side of human memory.
- "Append-only will blow up storage." View caching + periodic archival absorbs it. Events are mostly light JSON; trust beats storage cost.
- "We're single-session, state store is fine." Today single-session, tomorrow multi-session multi-agent. Pay the migration cost early.
If you're building an agent memory layer, run the six-point check above today — 30 minutes tells you whether your memory layer can ship to production.
I'm writing a "deep AI systems" series, one piece a week. Next: Agent task-queue state-machine design — why pending/running/done isn't enough.
If this was useful, an emoji react helps more people stuck in the same hole see it. Disagreements welcome in the comments — I'll reply to each.
Top comments (3)
The write side of this is right, and worth saying plainly: append-only with actor and rationale is the difference between a memory you can debug and one you can only believe. Two pushbacks from running a memory layer in production:
latest_wins moves the bug instead of fixing it. The moment the agent reads only the projected view, every disagreement in the log gets silently adjudicated by a timestamp - the same failure as a vector store letting 0.94 beat 0.91, relocated from retrieval to projection. The log knows "disliked X, then liked X, then a correction arrived"; the view says "likes X" with exactly the flat confidence your opening complains about. If the projection doesn't hand conflicts and supersessions back to the reasoning step, the trench-coat bug is still there - it just has better paperwork now. There's a 40-comment thread on precisely this trade-off under Edward Izgorodin's "Everything It Remembers Has the Same Authority" piece - your log/view split is the storage half of what that thread converged on, and the read side is where both halves still break.
A measured caution on "rationale must have a source" as a required field: fields the writer must remember get skipped or filled with noise. We counted our own store - 40 % of lessons (195 of 493) carried no author while it was a field an agent was supposed to send. The fix was deriving it from git config, not requiring it harder. If rationale can be derived from an artifact - the commit, the test run, the API response that triggered the write - it fills itself. If it's free text the agent must invent, the pending-adjudication queue becomes where writes go to die.
Curious whether this comes from a running system: which merge strategy beyond latest_wins have you actually needed in production?
Strong pushback, and the 40%-no-author number is the kind of evidence that changes a design. Two responses: On latest_wins: you're right that moving the adjudication into the projection just relocates the bug. The distinction I lean on is that the projection is a pure function of the log — so when latest_wins gets a case wrong, the log still has everything needed to re-derive the right answer under a different strategy, with zero writes lost. That's the property a state store doesn't give you; it's not "latest_wins is correct," it's "latest_wins is reversible." On rationale as a required field: your "derive it from an artifact" fix is the better answer, and it matches what I'd expect — a field the writer has to remember is a field the writer will fake. Requiring rationale harder pulls up fabricated text; deriving it from the commit / test run / API response makes it self-filling. I'll be stealing that framing. To your question: beyond latest_wins, the one I've actually needed is explicit supersede-by-actor — when a higher-trust actor (human review, a passing test) contradicts a model write, timestamp order is the wrong arbiter. Have you hit the case where a supersede and a later append disagree, and if so, which wins in cachly? (Also: the 40-comment thread you mention — dropping the link so I can read it.) 翻译
Late, sorry. Here's the thread: dev.to/izgorodin/your-agent-doesnt...
Which wins: the explicit one. Being later wins nothing by itself. A newer write that contradicts an older one without saying so doesn't overwrite or hide it. Both come back, and whoever reads them sees the disagreement. Only a write that names what it replaces moves the old one aside, and even then the old one stays readable with a pointer to its successor. Rewriting an existing entry needs a one-line reason, like a commit message.
We did test the timestamp idea, from the ranking side: older entries ranked lower. It broke every benchmark floor we have. Relevant lessons are often weeks old, and age says something about the note, not about whether the fact still holds.
Your supersede-by-actor is aimed at the part we haven't solved. Agents almost never say what they're correcting, because the agent writing the correction rarely has the old entry in front of it. A passing test or a human review does. They know exactly which belief they just overturned.