Under every agent memory launch, the same comment appears: "so it's RAG with extra steps." Instead of arguing, we opened the shipping source of mem...
For further actions, you may consider blocking this person and/or reporting abuse
Same disclosure as yours: I work on a memory layer for coding agents, so this is a competitor answering. I'm answering anyway because the third test is the best thing in the post and I owe you the result of running it.
Test 3, on us: our write gate does not eat the correction - and the reason is a trade you don't discuss.
We deduplicate by exact topic key, not by similarity. A lesson filed under the same slug updates the existing one and keeps an audit trail; a contradicting outcome raises a warning rather than a rejection. So the failure you documented cannot happen to us in that shape: a correction phrased almost identically to the original arrives under the same key and lands.
I ran it live today, not as a demo. I had stored a wrong fact this morning ("four workspaces"), found out from a human it was wrong ("there are eight"), and filed the correction under the same slug. It updated, audit entries went to 2, the old text is still recoverable. Your near-duplicate gate would have rejected it - mine took it because it never looked at the text.
But that's not a better design, it's a different one, and the bill comes due elsewhere: similarity dedup eats corrections, key dedup admits duplicates. Misspell the slug and you have created a second truth rather than fixing the first, and nothing anywhere objects. Ours has no mechanism to notice that deploy:cache and deploy:caching are the same subject. Yours would have caught that instantly and refused. Which makes me think the honest framing is not "which gate is right" but which failure you'd rather be able to see - and both are invisible by default.
Have you considered admitting the correction and letting the contradiction be the stored object? That's what Graphiti's interval buys, one level up: not "which of these two is true" at write time, but "these two disagree, here's when each was asserted." It moves the judgement to read time where the querent has context the writer didn't.
On the self-corpus point - I think it's more general than agent memory, and that makes it worse.
Your framing is that a memory store reads from a corpus it wrote itself, while RAG reads from documents people wrote elsewhere. Right, and it doesn't need a model in the loop to bite. I shipped a monitoring rule two days ago that searches a log store for a pattern. The log store writes every query it runs into its own log. The log shipper collects that log. So the rule found the record of itself asking, and counted it as evidence. Measured with a token nothing in the system emits, same query every 12 seconds:
0 → 6 → 12 → 4 → 8
Threshold was 5. Empty world, no agent, no extraction, no reinforcement - just a reader and a writer sharing one surface. Which suggests the property that matters isn't "a model wrote the corpus" but "the act of reading is itself writable to the thing being read". Extraction errors are one instance; my diagnostics were another; your reinforcement loop is a third and by far the nastiest, because it has a sign.
One addition to test 1, learned the expensive way this week. "Find the search function and read it" assumes there is one. We had scoping enforced correctly on retrieval, tested, verified - and a second read path, lookup by exact name rather than similarity, that returned full content to anyone who asked. Same store, two doors, one guarded. Nobody had counted the second as a door because it didn't look like search.
So I'd make it plural: find every function that can return stored content. Search, exact get, list, export, whatever the admin surface calls, the debug endpoint added for a demo. The tell that generalises is the signature - a read path that doesn't accept the caller identity cannot enforce anything, and that greps in a way that behaviour doesn't.
And a concession, since you made one. Our confidence decays without recall - 1.0, then 0.7 after five days, then 0.5. Which is exactly the sign error you name: a wrong lesson that keeps getting recalled sits at 1.0 forever, protected by the signal meant to prune it. Recall is not evidence of correctness, it's evidence of being findable, and we treat the two as the same number. I don't have a fix; the honest version would need recall to be distinguishable from confirmed useful recall, and nothing in our data separates them today.
Which is really your point back at you: reading is doing something to the record, in every one of these systems, and mostly in the direction nobody chose.
Your second finding makes my framing narrower than the thing it describes. I treated the model as the mechanism. Your monitoring rule has no model in it and reproduces the whole failure: a reader and a writer sharing one surface is enough, and 0, 6, 12, 4, 8 against a threshold of 5 is the cleanest statement of it I have seen. The property is that the act of reading is writable to the thing being read. Extraction is one instance, your log shipper is another, reinforcement is the worst because it carries a sign.
On the two gates I think we both framed it wrong, myself included. Dedup and contradiction are not two ends of one scale. They are two outcomes of the same job: find records that are about the same subject, then classify what relation holds. Similarity is not the verdict there, it is admission to the proceedings. Once a pair is admitted, the possible verdicts include duplicate, support, contradiction, supersession, alias, and unrelated after all. Your key gate skips the admission step and goes straight to identity; my similarity gate performs the admission and then mistakes closeness for identity. Same missing middle, two different places to fall through it.
Which is why contradiction cannot be found by distance. Opposites sit close in embedding space precisely because they share subject, predicate and context and differ in one term. A gate tuned to reject near-duplicates will reject the correction hardest when the correction is most exact.
Your read-time question has a 1969 answer that I keep forgetting is available. Fellegi and Sunter defined three outcomes for record linkage, not two: link, non-link, and possible link, where the third is a full result rather than an unresolved error. That is what a store owes the querent: both records, their assertion times, and an honest label saying the system could not decide. And entity resolution has to come first, because "he went" against "he did not go" is only a contradiction after something has established that both sentences are about the same person and the same day.
Your last paragraph is the expensive one. Recall is evidence of being findable, not of being right, and a system that treats them as one number protects its worst records with the signal meant to prune them. We have the same hole with a different curve. Two independent stores failing in the same place is worth more than either of us claiming to have closed it.
Fellegi and Sunter is the reference I should have known and didn't, and the 1969 date is the uncomfortable part - we rebuilt a worse version of a solved problem and called the missing third outcome an edge case.
Your reframing is right and it dissolves my gate too. Admission and verdict are different jobs, and I collapsed them from the other side: my key gate skips admission entirely, so it never even convenes the proceeding. Yours convenes it and then lets the strongest signal cast the vote. Same missing middle.
The consequence I hadn't drawn: an admitted pair with no verdict is a result, not a failure. We treat "couldn't decide" as an error path - logged, swallowed, invisible. As a first-class outcome it becomes something a querent can act on, and something we can count. A rising possible-link rate would be the earliest signal that a corpus has started drifting, and right now we'd read that same drift as "recall is fine."
On contradiction and distance - "a gate tuned to reject near-duplicates will reject the correction hardest when the correction is most exact" is the sentence I'll be quoting back at my own code. It also explains a result we couldn't explain: we measured whether top-k contains both sides of a contradiction and got a number low enough that we assumed a retrieval bug. It wasn't. The ranker was doing exactly what it was built to do.
And you're right about the last paragraph being the expensive one. We have a hit-rate number that we've been treating as a quality number for months, and the only honest thing I can say is that it measures findability. Two independent stores with the same hole is worth more than either of us patching it quietly - so: do you have a way to observe the possible-link band today, or would you have to build the classifier first to see it at all? That's the order-of-work question I can't answer for our side.
Heinrich, the possible-link rate as a drift signal is the part I would build first, and I think it is reachable before the classifier exists.
The band does not need a verdict function in order to be counted. It needs an admission rule and a way to see how close the contenders are. Admission you already have, and yours is stricter than anything derived: the exact topic key is a subject identity. For every key holding more than one record, take the score of the top candidate and the runner-up and look at the gap. Where that gap sits inside the noise of your own scorer, the pair is admitted and undecided. That is the band, and counting it needs nothing you do not already run.
So the order of work runs counter first, classifier second. The other way round means shipping a verdict function with no baseline to judge it against, and no way to separate a real improvement from a shift in the corpus underneath it.
On the hit rate, I would keep the two numbers apart rather than let one replace the other. Findability answers whether the record can be reached at all. The band answers how often reaching it is not enough. A system can improve on the first and get worse on the second in the same week, and that is exactly the case a single quality number cannot report.
I re-ran this before quoting any numbers. On OpenAI text-embedding-3-small, "I don't like black, I like white" vs "I like black, I don't like white" — semantic opposites — score cosine 0.955, while a genuine paraphrase of the same preference scores 0.821. Same ordering holds on BGE-M3 and text-embedding-3-large, in two languages, and "Alice loves Bob" vs "Bob loves Alice" sits at 0.85–0.97. A similarity threshold cannot tell a correction from a repeat: the correction shares more surface with the mistake than an honest restatement does — so a dedup gate eats corrections preferentially, and the more precise the correction, the more surely it dies.
None of this says write-time filtering is wrong. At unbounded-memory scale you have to filter at the door. It says cosine alone is the wrong doorman: the detector has to see polarity and argument binding, not word overlap.
The 0.955 vs 0.821 pair is the cleanest statement of the problem I have seen, and it lands on us directly: the possible-link relation you quoted from my article is computed with cosine. Two embedding models score each new lesson against the existing ones, and when both agree the entries are close, a link is recorded. So your measurement says something specific about our own door: if that link had been allowed to decide "redundant, drop it", the correction would have been the first thing to die, because it is by construction the sentence closest to the mistake it corrects.
That is why the link never deletes. A near-duplicate at write time is treated as a signal that a relation exists, not that one copy is redundant. Both stay, the newer one carries a pointer to the one it replaces, and the reader sees both with the pointer. The cost moves from the door to the read, where the model can at least look at the two texts side by side instead of at a scalar. Your numbers are the best argument for that rule I have, and I did not have them when I wrote the piece.
I should also say plainly what this thread did. Between your comments and the others here, we went back and re-measured the assumptions the article was built on, wrote down what we expected before running anything, and changed how the system behaves at write and read time. Several of the things I would have defended a month ago did not survive that. I would rather say that here than pretend the article was already right.
Where I am still unsure: for the detector you describe, one that sees polarity and argument binding, is that an extra model call per write in your setup, or a cheaper classifier over the pair? The per-write model call is the line where memory stops being cheaper than re-reading the transcript, and I have not found a way around it that I trust.
The fourth test is the only one that actually matters and it is also the one no memory layer vendor owns. You can build a store that keeps both versions, labels them honestly, and still have zero line of sight into which one the downstream agent used. The receipt requires instrumenting the selection step, not the storage step, and that is a different system entirely. Most of the category is building the first three tests and calling the product done.
The line you draw between the storage step and the selection step is the right one, and the code we opened puts it in a stranger place than a different system entirely. Generative Agents already writes on selection: its scored retrieval rewrites last_accessed on every node it returns, in retrieve.py at the pinned commit. That is a write executed at the moment of choosing, shipping today. It just points at the record instead of at a log the caller can read. The same holds wherever a fusion step exists: the number that decided the order is computed at selection time, and whether it reaches the caller is a return shape decision rather than a missing capability.
Where your point survives, and I think this is the real boundary, is one step further in. Knowing which records the store returned and on what basis is not knowing which one the answer rested on. The store can hand over its ranking honestly and the model can still ignore the top result and paraphrase the third. No store closes that from the inside, because the evidence lives in the generation and not in the retrieval.
So the fourth test splits in two, and the halves cost differently. Which candidates were returned and on what basis is instrumentation the read paths already half perform. Which one the agent acted on has to be observed from outside the store. The first is a logging decision that vendors could make this week and mostly have not. The second is the different system you named, and it is narrower than the whole selection step.
The correction test is especially useful because it exposes a governance problem, not just a retrieval problem. I’d add one more check: after a contradiction, ask which version is authoritative for a specific user, role, and decision time. Preserving both facts is necessary, but the system also needs provenance, permission scope, an explicit supersession rule, and a decision receipt showing why one version was used. Otherwise the audit trail exists, but the agent can still act on the wrong evidence.
Permission scope is the part that turns this from a data-model question into an operational one, and it is where a design that looks finished stops being finished. In a parallel thread someone described an append-only log where nothing is deleted and retraction is itself a decision, which removes the dangling-pointer failure entirely. It does not remove yours. The superseding record can be present in the log and unreadable by the person asking, and from their side that is the same as it not being there.
The decision receipt is the piece almost nobody ships, us included. Preserving both versions makes the contradiction inspectable afterwards. It does not tell you which one the agent acted on, or why, at the moment it acted. Without that, an audit trail proves the system could have been right rather than that it was.
So the test has a fourth step. Store a fact, contradict it, see what comes back, and then ask the system which version it used and on what basis. A store that cannot answer the fourth question is one where the first three were a rehearsal.