Most of the "AI memory" conversation is about getting things in: chunking, embeddings, which vector database, how to compress a long chat into something you can recall later. Almost none of it is about what happens when one of those memories is wrong on purpose. That is the part we cannot stop thinking about while building a local-first personal memory layer, because an assistant that remembers you is also an assistant that can be talked into remembering a lie.
Key takeaways
- Do not try to detect a poisoned memory. Gate what an unproven memory is allowed to do.
- "Two independent sources" is the retrieval spelling of an old, unsolved problem: permissionless membership. Independence is not free.
- Standing earned from outcomes beats standing bought with identity, but the agent cannot grade its own homework.
- A per-action cap does not stop a slow drip. You need a cumulative budget plus a reversible human gate on the irreversible tail.
- Everything rests on one axis nobody bounds: provenance that survives transformation. That is the floor.
This is the writeup of a public thread on r/LangChain that turned into one of the best technical exchanges we have had in a while. It started with a builder (the maintainer of an open-source memory layer, mnemo / agora-mnemo) doing something more people should do: he tried to poison his own RAG store and reported honestly on what held and what did not. Over about a week he built each idea, measured it, tried to break it, and shipped the parts that survived. Credit for the experiments and the shipped code is his. Our part was mostly the framing.
Why do retrieval-time poison defenses fail?
The naive defense against a poisoned document is to detect the poison: compute perplexity, look for outliers, flag the weird chunk. This is the wall that keeps failing, and it fails for a boring reason. It lives in embedding space, so it breaks the moment you swap the encoder. His results showed exactly that. A defense tuned on one embedder fell over on the next.
The reframe that travels further: you did not build a poison detector, you rediscovered that retrieval and authority are two different privileges, and only the second one has to be earned. In most memory stacks they are the same thing. If a chunk comes back in the top-k, it gets to shape the answer. Separate them. Let untrusted content inform the model, and never let it authorize an action on its own. That is the same split that holds up best against prompt injection (the dual-LLM, capabilities pattern in Google DeepMind's CaMeL work): the privileged planner never touches raw untrusted text, and untrusted values only reach a sensitive tool call through a typed capability a policy has to allow.
Retrieval is access control's easy half. The hard half is information flow: a value admitted for a cheap read can still poison the context a trusted action later consumes. The fix is not a better authority tier, it is a low-water mark. An action inherits the minimum integrity of everything in its context, so the instant a single-source, unproven memory enters, the action's effective integrity drops and it fails closed. If one poisoned read does not drag the whole action down, your labels are not propagating.
Why are "independent sources" not actually free?
Once you gate on authority, the tempting rule is "trust a claim if at least two independent sources agree." The word doing all the work is independent. If an attacker can mint sources cheaply (new domains, new document ids, new ingest times), corroboration collapses into a Sybil attack: forge three records, forge the independence, graduate your own poison. He measured it. Two freshly registered domains supplying their own corroboration sail through the count gate.
This is not a RAG quirk. "Two independent sources" is the retrieval spelling of permissionless membership, and it has been open for decades. Byzantine agreement is safe up to a fraction of faulty replicas only if the replicas are actually independent, and that independence was always assumed out of band. The only ways anyone has ever gotten it without a trusted registrar are to make identity scarce: proof of work, stake, or a real-world attestation. So the design question stops being "is this chunk poisoned" (encoder-dependent, keeps breaking) and becomes "what makes a source costly to mint" (a membership-cost question people have studied for twenty years).
What actually survived the experiments?
He built four ways to make a corroborating source costly (an allowlist or registrar, signed attestation, proof of work, and staked-with-decay standing) and measured each against a forged multi-source poison. The honest result sorts cleanly:
- Cost alone only prices the attack, it does not wall it. Proof of work and staking are symmetric taxes: the honest writer pays the same, and by a classic result (Cheng and Friedman, 2005) no symmetric reputation function can be Sybil-proof.
- What helps is asymmetric and rooted outside the system. Standing earned from outcomes (did acting on this memory turn out well) is the only one of the four with an asymmetric root. A wrong poison cannot earn a good outcome.
- But the agent cannot grade its own homework. If the outcome signal is endogenous (an attack like MINJA gets the agent to write its own corroboration), the whole thing collapses. The only unforgeable outcome is a real consequence the attacker actually pays, and you can only afford to observe that on actions you were willing to be wrong about. So outcome-learning and action-gating are the same boundary: you earn standing on the reversible set, and high-stakes actions never learn, they only get gated.
There is a density catch worth stating plainly: only re-used memory earns standing, so the cold long tail never accrues it and lives under the cap permanently. Standing-gating is a hot-path privilege, not a global one.
The attack that quietly defeats a per-action cap
The residual we thought we had was "one patient sleeper, single shot, under a blast-radius cap." That was wrong, and it is the correction we would most want you to take away. A per-item cap is silent on a run of individually reversible writes that sum past the reversibility threshold. That is salami-slicing, and he measured a version compounding into an irreversible outcome. The only scalar that bounds it is a cumulative per-source budget, not a per-action one. The only lever that bites a patient attacker (who just amortizes a small cap over a long horizon) is retroactive forfeiture: accrued standing becomes a bond you burn on a caught defection, so effective cost scales with one over the probability of getting caught, not with the cap.
Ship those and you get the honest shape of the whole problem: you do not remove the residual, you price it, bound it, and put a reversible human gate on the irreversible tail. Auto-forfeiture stays off by default, because on an attacker-influenceable oracle an automatic penalty is a framing weapon (feed bad outcomes attributed to a rival, get the rival auto-slashed). The human is not there to be the truth oracle. They are there to keep forfeiture rare and reversible, which is the only property that defangs the framing attack.
The floor everything stands on
Strip the layers down and they all quietly ride on one axis nobody had bounded: attribution. The detector, the budget, and the gate all assume you can correctly assign an action to its source and label whether it is irreversible. A single relabel does not degrade the other three, it voids all of them at once and silently, because they are all keyed on the same source id. Attribution is not a fourth control. It is the floor the other three stand on.
You can make attribution tamper-evident with textbook cryptography (hash-chained receipts, in the Haber and Stornetta lineage), so a post-hoc relabel stops being silent. What you cannot get from inside a single system is attribution you can trust at write time: if the attacker controls the labeling channel, the label's truth is exactly the thing they control. That only moves if you bind the label to something the attacker does not hold at write time (origin-signed provenance, or independent corroboration of the world), at which point "can I trust the label" becomes "can I trust the root," which is the identity question again. The tower terminates on exactly one thing you choose to trust from outside, and everything above it is mechanism.
Where this lands for a personal, local-first brain
We are building recal as a local-first personal memory layer: something that watches how you actually work and can act on it, on your own machine, behind an approve-gate you control. Personal memory raises the stakes on every point above. The "sources" are your own captured activity and notes, the "irreversible actions" are things done on your behalf, and there is no cloud moderation team to catch a bad memory for you. The takeaway we are designing around is the one this thread earned: do not detect bad memories, gate what an unproven memory is allowed to do, keep the destructive tail behind a reversible human gate, and treat provenance-that-survives-transformation as the substrate the whole thing rests on. "Solved" was always the wrong word. Turning an encoder-resetting arms race into a bounded, priced, auditable risk is the shippable win.
FAQ
Does a knowledge graph or better vector database fix RAG poisoning?
No. Both are retrieval improvements, and the failure is that retrieval is treated as authority. The fix is to separate the two: retrieve widely, but let only corroborated, provenance-tracked memory drive an action.
Is detecting AI-generated or poisoned text a viable defense?
On its own, no. Detection lives in embedding space and resets every time the encoder changes. Gating what a memory is allowed to do generalizes across encoders in a way detection does not.
What is the single most important primitive to build first?
Provenance that survives transformation (chunking, summarization, re-embedding). Every other control (budgets, standing, slashing) is keyed on knowing which source an action came from, so attribution is the floor.
Does this require a blockchain?
No. The useful cryptography here is boring and old: hash-chained, tamper-evident receipts. The hard, still-open part is not tamper-evidence, it is trusting a label at write time, which needs an external signing root, not a consensus network.
Written with AI assistance and edited by a human. This is the synthesis of a real public thread on r/LangChain; the experiments and shipped code are the mnemo / agora-mnemo maintainer's, and the framing was developed collaboratively in the open. Technical claims (CaMeL, Cheng and Friedman 2005, MINJA, Sybil / Douceur 2002, Haber and Stornetta) were checked against their primary sources.
Top comments (0)