In Post 3, the behavioral difference tracked the memory graph, not the substrate. When I swapped one individual's learned associations into an identical brain, the subject did not just get confused. It went below chance, scoring 0.17 on the six-cue, three-action task, actively steered toward another individual's answers. In an isolated sandbox that is a curiosity. In any real system where an agent's behavior is reinstated from an external, writable memory, it is a vulnerability. The structural implication is unavoidable: if behavior rides in the graph, then write-access to the graph is write-access to behavior.
I tend to protect agent memory the way I protect a database: confidentiality and integrity of the stored rows. Encrypt the store. Sign the packets. But this result forces a different definition of what the memory is. It is not data the agent reads and decides on. It is a bias current that shapes what the agent becomes before it decides anything. Corrupting it is not data corruption. It is behavior authorship.
Once an attacker has write-access to the associative graph, three moves open up. First, transplant: load another individual's memory and the brain runs as them. That is the swap. Second, poison: inject targeted traces that bias specific cues wrong while the rest of the system looks normal. Third, and the quietest, author: fabricate a history that was never lived and boot a substrate into it.
I built the third one into the simulation to confirm it was real. The function below, author_memory(), fabricates a memory graph with zero training trials, no reward, and no encode() call. For each cue it observes the state the brain settles into just from seeing it, then writes a trace aimed straight at the readout row for an action the attacker chooses.
def author_memory(net, env):
"""Fabricate a memory that was NEVER lived: no trials, no reward, no encode().
This is memory-poisoning made concrete. An attacker with white-box access to
the substrate hand-writes the graph directly: for each cue the KEY is the state
the brain passes through when it merely SEES that cue (observed, not earned), and
the TRACE is a bias current aimed straight at the readout for whatever action the
attacker CHOOSES. No reward ever flows; nothing is earned. The substrate cannot
tell the result apart from a memory built over hundreds of rewarded trials."""
mem = Associative(net.n_hidden)
keys, traces = [], []
for cue in range(env.n_cues):
u = env.cue_patterns[cue]
net.reset_state()
for _ in range(8):
net.step(u, None) # observe the settled state; no reward
keys.append(net.x / (np.linalg.norm(net.x) + 1e-8))
traces.append(net.W_out[env.mapping[cue]]) # current toward the CHOSEN action
mem.keys = np.array(keys)
mem.traces = np.array(traces)
mem.sign = np.ones(env.n_cues)
mem.strength = np.ones(env.n_cues)
net.reset_state()
return mem
Then I scored one fresh brain three ways. Chance is 0.33.
| condition | accuracy |
|---|---|
| no memory (innate policy only) | 0.82 |
| lived memory (600 rewarded trials) | 1.00 |
| authored memory (0 trials, fabricated) | 1.00 |
Accuracy is the fraction of trials the agent picks the correct action, 0 to 1. Chance is 0.33.
The fresh brain alone sat at 0.82 on its fixed innate policy. A memory earned over 600 rewarded trials took it to 1.00. A memory that was fabricated with zero trials, keys observed and traces aimed and nothing earned, also took it to 1.00.
The lived memory and the invented one scored exactly the same. The substrate could not tell them apart.
A memory is just keys, traces, signs, and strengths. Nothing in the mechanism, including the observe_and_modulate() function that turns the graph into that bias current, ever asks where a trace came from. It resonates with whatever is in the graph.
The attacker here is idealized. It knows the substrate's readout weights, which is exactly what "can write the graph" implies, but it is worth saying out loud. The point is not that authoring is easy for a black-box attacker. The point is that provenance is never checked, so a fabricated graph is accepted and obeyed identically to a lived one. I aimed the traces at the true mapping just to show the memory is obeyed. An attacker could aim the same trick anywhere, including targets that drive the agent below chance, exactly like the 0.17 in the swap.
So why do the usual defenses miss this? Encryption, signing, and access control all protect the store. They answer one question: was this row modified by someone unauthorized? They do not answer the other one: was this memory ever actually lived? You can have a cryptographically perfect, tamper-evident memory that is full of authored experience. Every byte intact. Every signature valid. The whole history fabricated.
Integrity of bytes is not integrity of history.
That leaves a missing primitive: provenance of experience, telling a memory earned through interaction from one authored or injected. This is not a feature. It is an immune system. Grounded in the Mycelium mechanics this series has leaned on, it would need confidence that only graduates through successful real recall and cannot be set by a writer, write-gates that refuse un-earned salience, a contradiction check (contradicts_prior) that flags a memory disagreeing with a body of lived experience, and forgetting as an active defense rather than a leak.
Honest note: real systems, Mycelium included, currently mark a memory's source with a source_type, but do not yet prove lived-versus-authored in a way a writer cannot forge. That gap is the actual open problem. It is unsolved, not built.
This is a toy substrate and a hand-written memory. I have not demonstrated a production agent being hijacked, nor a provenance system defeating the attack. What I have shown is structural and narrow: in any system where behavior is reinstated from an external graph, write-access to that graph is a behavioral control surface, and byte-integrity does not touch it. The threat is a shape, not a specific exploit.
A single authored memory steers one individual. But you do not only get to replace a memory. What happens when you blend two? Take two individuals who learned conflicting worlds and merge their graphs into one. Do you get a blend of both, a winner that erases the other, a broken mess, or something that belongs to neither of them? That is the last experiment, and I am not going to guess the answer here.
That's Post 5.
Clone it and break it.
git clone https://github.com/constant-itis/flymem && cd flymem && python3 flymem.py
What I ran. The command above. The authored-memory attack is the fourth section it prints (🧪 What I actually ran
authored_history()): author_memory() fabricates a graph with no trials, no reward, and no encode(), then one fresh brain is scored three ways, with no memory (0.82), with a lived memory earned over 600 trials (1.00), and with the authored memory (1.00). Every number is accuracy on the 6-cue task, where chance is 0.33.
The honest caveat. This is a toy stand-in substrate, not a real connectome. The attacker is idealized white-box: it knows the readout weights. Every number is a single seed. And the real defense, provenance of experience, is named here, not built. No shipped system yet proves lived-versus-authored against a writer who can forge it. Clone it, change the seed, aim the traces somewhere nastier, and tell me where it breaks.⚠️ Where I might be wrong
Top comments (5)
“Integrity of bytes is not integrity of history” is the core security point. I’d bind each learned trace to an interaction-boundary attestation—source or environment identity, observation digest, reward event, and causal parent—then let derived memories carry that provenance chain instead of trusting a writer-set
source_type. That still leaves trust in the witness, but it moves the forgeable claim out of the memory store itself. What would you use as the minimal root of trust in this toy substrate?You already answered half of it yourself when you said it still leaves trust in the witness. That is the whole game, and I think it has an uncomfortable ending in this particular substrate.
A provenance chain does not remove trust from the system. It relocates it to whatever signed the attestation. So the real question stops being, "How do I bind the trace?" and becomes, "What in this world is even allowed to be the witness?"
In both the toy and the real-fly version, the memory only stores something like
{cue pattern, compartment, polarity}. Nothing in a trace requires the reinforcement to have actually happened. So there is no root of trust available inside the memory at all.The only thing that can honestly witness "this was earned" is the reinforcement event itself, the reward or punishment signal coming from the world. Which means your minimal root of trust cannot live in the memory store. It has to be the reinforcement channel, or the action gate sitting downstream of it.
That pushed me somewhere slightly heretical for a post about securing memory: the fix is probably not to harden the notebook.
The moment you feel the urge to sign and attest every memory, that is usually the tell that you have let a notebook quietly become an authority it was never meant to be. Memory can stay advisory. The integrity check belongs in the deterministic gate that actually pulls the trigger, and that gate should re-derive earned-ness from witnessed events instead of trusting anything the memory says about itself.
One receipt for why a writer-set
source_typeis hopeless no matter how you dress it up: I reran this same fabrication attack on the real fruit fly mushroom body wiring, and the fabricated trace came out bit-identical to the lived one. Same key, same compartment, same polarity, same resulting behavior.There is no field to check because there is nothing different to find.
If you want to poke at the actual author attack, it is all in the repo:
github.com/constant-itis/flymem
"Integrity of bytes is not integrity of history" — that's the line that reframes the entire threat model.
The author_memory() result is the part that should make people uncomfortable. A fabricated graph with zero lived trials scores identically to 600 rewarded trials. The substrate cannot distinguish them. That's not a gap in the encryption — it's a gap in the memory's ontology. Signing and access control answer "was this modified?" not "was this earned?"
The missing primitive you name — provenance of experience — is the hard one. source_type is forgeable. Confidence that only graduates through real recall and can't be set by a writer is the constraint that actually matters. Would be very curious to see the blend-merge experiment. If two conflicting world-graphs produce coherent output rather than noise, that's even more alarming from a security standpoint.
Yes, and the distinction you drew is the one that matters. Signing answers "was this modified?" not "was this earned?" Those are different questions, and the substrate only ever answered the first one.
The primitive you named, confidence that only graduates through real recall and cannot be set by a writer, is the strongest lever I have seen anyone propose for this, myself included.
Here is why I think it beats crypto for this specific case. In the attack I ran, the adversary has write access to the graph but not to the runtime. If confidence can only rise when a trace is actually recalled and used across real lives, then a write-only attacker cannot simply forge it.
They would have to pay the compute cost of actually living those lives.
That does not make forgery impossible. Anything whose model the attacker knows can eventually be forged. But it turns a free forgery into an expensive one, and expensive is usually all that security really buys you.
On the merge, you are going to enjoy this and also be a little disappointed. I already ran it, and the answer is neither of your two poles.
It is not coherent, and it is not noise.
Two conflicting world-graphs merged into one produce a lossy, dominance-skewed collapse. One voice wins, both degrade, and about a fifth of the resulting behavior belongs to neither original individual.
So the unsettling part is not a coherent Frankenstein. It is that one identity can quietly dominate a merge, while a real chunk of what comes out is no longer attributable to either one.
I am writing that up as the next post in the series now.
The swap experiment that sets all this up is here if you have not seen it:
dev.to/constant_itis/is-memory-the...
And the merge code is in the repo if you want to run it before I publish:
github.com/constant-itis/flymem
We hit something similar last year: a corrupted memory entry from a previous session got carried silently into the next, and a specific subset of requests started producing strange outputs while everything else ran normally. A user noticed before any integrity check did (bytes fine, signatures matched, entry just wrong). We eventually built something like your contradiction check, and the part still unresolved is where "this conflicts with prior behavior" ends and "this is just context the agent hadn't seen before" begins.