Special feature of [Building Your Own Agent] series · All engineering practices come from the open-source project ResceneAgent
Let me give you the answer in one sentence:
An agent's memory is not a warehouse that keeps piling up. It's a web that grows, forgets, and gets pruned.
Every time you tell it something, it ties a knot on the thread. When two things are related, another thread connects the knots. Over time, the web grows denser — it understands you better, but it can also grow slower.
Because the hard part has never been "storing things in." It's three things:
- What should be remembered?
- How do you find it when you need it?
- How do you let go when it's obsolete?
Borges' Funes remembered everything, and it made it harder for him to think. When every leaf and every cloud pattern is equally vivid, you can't extract patterns from the details. The same is true for an agent — a memory that never forgets isn't wisdom; it's more like a hard drive failure.
Turing, in his 1950 paper Computing Machinery and Intelligence, proposed the idea of a "child machine": instead of building a fully grown adult brain from scratch, start with a simpler foundation and let it grow through education. What we're doing today when we build memory systems for agents is, in a sense, answering that seven-decade-old question: how can the things a machine has experienced become the experience it brings to its next action?
To tell this story properly, we need to take a small detour — starting with HTML. Then we'll look at a very special electronic component — the memristor.
And this isn't just a survey of approaches. Later, I'll walk through a real two-month experiment: I built a fully functional "digital hippocampus" that could spread activation, decay over time, and even let you watch memory being recalled hop by hop. Then I ran a controlled experiment and proved, with my own hands, that it wasn't worth keeping on the mainline.
Building a system is skill. Killing your own pride and joy — that's engineering judgment.
I. From HTML to Markdown: Letting Knowledge Hold Hands
In the summer of 1991, Tim Berners-Lee opened the early Web system to the wider community. The most magical thing about HTML wasn't headings, paragraphs, or tables — it was the link:
<a href="another-page.html">Go to another page</a>
In the paper world, a page ends when you finish writing it. On the Web, one document can reach out a hand and hold another.
Berners-Lee later recalled that his dream for the Web was a shared information space: links could point to anything, whether it belonged to an individual or the public, whether it was a draft or a finished piece.
This change may look like just one more tag, but in reality it was like building roads for knowledge. Documents used to be isolated islands. With hyperlinks, bridges were built between them for the first time.
But HTML was designed for web page structure and browser rendering. If all you want to note down is "the user prefers short replies," having to write:
<html>
<body>
<p>用户喜欢简短回复</p>
</body>
</html>
It's like building an archive just to jot down a grocery list.
In 2004, John Gruber released Markdown. Its core philosophy wasn't "more features" — it was "fewer symbols": raw text should be natural, clear, and readable even without rendering.
# Title
- One record
- Another record
[Link to another document](another.md)
A # is a heading, a - is a list item, and a pair of brackets makes a link. As for the [[wikilinks]] common in knowledge base software — strictly speaking, they're not part of the original Markdown spec, but extensions added later by wiki and note-taking tools. They carry the same spirit though: make connections between knowledge light enough that humans will write them without thinking, and machines can read them without effort.
This is exactly why Markdown works as a carrier for agent memory: transparent, editable, easy to version-control, and both humans and models see the same content.
But file formats only solve "where to store it." The harder question is: once the memory is there, how do you find it?
II. The Memristor: A Resistor That Remembers Current Has Flowed Through It
An ordinary resistor is like a door with no memory: you push it a hundred times, and the next time it's still the same.
A memristor is different. Its name comes from memory resistor. The charge that flows through it changes its internal state, so the next time current passes through, its conductance is affected by its past experience.
Think of it as a dirt path:
- When nobody walks it, grass slowly grows back, and the path becomes harder to find.
- The more people walk it, the more the earth gets packed down, and the clearer the path becomes.
- The next time you come to a fork, you naturally take the well-worn path.
Isn't this exactly what memory is? Connections that are repeatedly recalled grow stronger; connections that go unused gradually weaken.
In 1971, Leon Chua derived the memristor — the "missing circuit element" — from the symmetry relations between circuit variables. In 2008, Strukov et al. published The Missing Memristor Found in Nature, giving the physical model that became widely known. What attracted neuromorphic computing researchers to the memristor was precisely that its conductance retains history: a single device can both "store" and exhibit effects similar to synaptic weight changes.
Why I Put It Into a Memory Engine
The early version of Prism had a C++17 vector storage prototype:
Vector Storage
+ HNSW Nearest Neighbor Search
+ Memristor State
+ Chaotic Evolution
The idea was compelling: don't make the vector store a row of unchanging drawers. Instead, let each memory behave like an electronic synapse — with conductance, flux, and evolving state. An access doesn't just "find it"; it changes it. The passage of time doesn't just add a timestamp; it lets the memory drift slowly.
In other words, a normal vector store is like a map: once the roads are drawn, they stay there. The memristor model is more like a real city: foot traffic turns small paths into main roads, and abandoned roads get swallowed by weeds.
I ended up cutting this idea. Not because it wasn't cool, and not because it didn't work — but because it pushed the system into a different kind of complexity: the C++ vector layer, C API, Go service, chaotic state, index structure, and upper-level memory graph all had to evolve together. In trying to simulate "being like a brain," the engineering itself grew into a brain that was hard to maintain.
But the memristor left behind its most important legacy:
Memory shouldn't just be stored. Every use should change the probability that it will be recalled next time.
Later, when I rebuilt PrismD in Go, I dropped the C++ prototype but kept this idea: node energy decays over time, and access strengthens it. Even the synapses connecting two memories have their own decay rate. The hardware metaphor was removed, but the useful dynamics remained.
This was also the first time I truly understood: subtraction isn't about deleting everything. It's about removing the expensive form and keeping the effective principle.---
III. Four Memory Engines: From Flipping Through Books to Weaving a Web
The market is full of names for agent memory solutions: vector stores, RAG, knowledge graphs, semantic networks, long-term memory, episodic memory... The terminology is like a smoke screen.
Strip away the labels, and the common implementations roughly fall into four approaches.
1. Bolt-on RAG: Giving Your Agent a Librarian
RAG stands for Retrieval-Augmented Generation.
It doesn't stuff the entire library into the model's brain. Instead, it sends a retriever to the shelves to find the few most relevant pages, then hands those pages to the model to answer the question.
User asks: "How did we fix that login function last time?"
↓
Retriever searches the document store for relevant fragments
↓
The matched fragments are handed to the LLM
↓
LLM generates the answer combining the question and retrieved material
In 2020, Lewis et al. described RAG as a combination of parametric and non-parametric memory: the model's parameters are like "knowledge it has memorized," while the external index is like a reference library it can consult at any time.
So describing RAG as "re-reading your entire diary from scratch every time" isn't quite accurate. A better metaphor: before you speak, you ask a librarian to fetch a few pages from the archive based on keywords, semantics, or metadata.
Its advantages are clear: material can be updated anytime, and the knowledge base can be large. The downside is equally obvious: if the retrieval picks the wrong shelf, no amount of clever answering can fix it. RAG is more "query-on-demand" than a continuous understanding of the user.
2. Graph Diffusion: From One Streetlight, Lighting Up the Whole Block
The second approach models memory as a graph: each memory is a node, and nodes are connected by relationships.
[Transformer]
/ \
/ \
[Attention] —— [GPT]
\ /
\ /
[Scaled Dot-Product]
When the "Attention" node is activated, energy spreads along the edges, waking up related concepts like Transformer, GPT, and Scaled Dot-Product. This is similar to the "spreading activation" theory of semantic memory proposed by Collins and Loftus in 1975: once a concept is triggered, activation propagates along the associative network.
It's like a city at night. You light up one streetlamp, and the current travels along the road, gradually illuminating the surrounding streets.
The benefit is that it can discover indirect relationships: A isn't directly connected to C, but A connects to B, and B connects to C, so the system can still find its way.
The cost comes from the web itself: how to deduplicate nodes, how to weight edges, how many hops activation should travel, how to decay old relationships, how to repair a damaged graph... Once the scale grows, you're no longer maintaining a notebook — you're maintaining a city's transportation system.
3. Pure Markdown: Giving Your Agent a Box of Index Cards
The third approach is the simplest: memory is just Markdown files. Load them when needed, skip them when not.
memory/
├── index.md ← Lightweight directory: links + one-line summaries
├── preferences.md ← User preferences
└── project-rescene.md← Project knowledge
A typical workflow:
Task: "Change the login page"
↓
Read index.md, hit [[project-rescene]]
↓
Read the relevant memories in project-rescene.md
↓
Inject into context, agent starts working
If RAG is a librarian, pure Markdown is a box of index cards. You don't need a database console, and you don't need to guess what's inside a black box. Open the file, and everything the agent remembers is right there. Mistakes can be fixed, outdated content can be deleted, and version changes can be tracked by Git.
Its weakness is equally straightforward: links usually take you to the next card, but they don't automatically do complex multi-hop reasoning. With a small number of cards, it's wonderfully light. When the cards fill a whole room, finding the right one becomes a new problem.
4. Structured Markdown: Stamping a "Confidence" Seal on Every Card
Structured MD doesn't abandon Markdown. It adds a few more fields to each card: source, type, confidence level, last updated, scope of applicability.
- Content: User prefers short replies
Confidence: High
Source: Confirmed over multiple conversations
Last updated: 2026-08-02
- Content: Project may use Vue 3
Confidence: Pending verification
Source: Single mention by user
Pure Markdown is like sticky notes. Structured Markdown is like putting those sticky notes on a whiteboard and marking them with different colors: "confirmed," "pending verification," "possibly expired."
This matters because memory doesn't just go missing — it can also lie. Not maliciously, but by mistaking a one-time event for a pattern, treating yesterday's fact as today's truth, or confusing the model's own guess with something the user actually said.
Metadata is the "nutrition label" for memory: it tells the agent where this information came from, how long it's good for, and whether it's safe to use.
Of course, the finer the labels, the higher the maintenance cost. If you also ask the LLM to periodically merge, deduplicate, depreciate, and retire memories, you've hired an archivist: the room is tidier, but the archivist expects a salary and might misfile things.
IV. Putting the Four Approaches on the Same Table
| Approach | Most Like | Advantages | Main Costs |
|---|---|---|---|
| Bolt-on RAG | On-call librarian | Large capacity, fast updates, good for external knowledge | Heavily dependent on retrieval quality; may not form continuous user memory |
| Graph Diffusion | City that lights up along roads | Can discover multi-hop and indirect relationships | Graph structure, edge weights, decay, and maintenance are all more complex |
| Pure Markdown | Transparent box of index cards | Simple, readable, editable, easy to version-control | Limited associative ability; becomes hard to search at scale |
| Structured Markdown | Archive cards with source and expiration date | Can express confidence, time, and scope | Requires additional organization and maintenance mechanisms |
There's no "the more technically sophisticated, the better" here.
Building a knowledge graph for ten preferences is like building a跨海大桥 just to cross a small stream. Stuffing millions of documents into Markdown is like managing a national library with sticky notes.
There is no single answer for memory engines — only the answer that matches your scale, your task, and your maintenance capacity.---
V. My Crucial Experiment: How I Sentenced My Own "Digital Hippocampus" to Death
If you only saw the final solution, you'd think I chose Markdown because graph engines were too hard, or PrismD never worked.
The opposite is true: PrismD was sentenced to death after it was fully operational.
1. I Really Built a "Digital Hippocampus"
PrismD modeled memory as a weighted directed graph:
- Node: A memory, with text, emotion, importance, and energy;
- Synapse: Associations between memories, categorized as associative, temporal, semantic, and episodic;
- Cluster: Logical zones — user profiles, code work, tool logs, sessions;
- Domain: Physically isolated spaces for different users or roles.
Its memory lifecycle wasn't simple CRUD. It was more like biological metabolism:
ENGRAM write
↓
DRIFT decay
↓
LOOM recall and strengthen
↓
COMPILE compress
↓
CONSOLIDATE merge / discard
↓
PRUNE active forgetting
Even synapses could forget:
Effective Weight = Initial Weight × exp(-decay_rate × time_since_last_use)
A connection left unused for too long rusts like an abandoned railway line. Each time it's used, it proves it still has value.
2. The Hard Part Wasn't "Association" — It Was "Cross-Contamination"
Graph diffusion makes it easy to build a stunning demo: light up "first love," and it wakes up "rainy day," "train station," "that song." But the more dangerous problem in production is: things that shouldn't be associated also crawl along the edges.
For example, a tool error shouldn't pollute user profiles. A temporary session shouldn't contaminate long-term project knowledge.
So I built an explicit inter-cluster propagation matrix:
ToolLog → UserBase = 0.05
CodeWork → UserBase = 0.8
UserBase → Session = 1.0
When a tool log propagates to the user profile, its energy is compressed to 5% of the original. This isn't a post-retrieval patch filter — it's writing "memory boundaries" directly into the propagation dynamics.
This lesson is crucial: A memory system needs not just recall rate, but also contamination prevention. Remembering something wrong is often more dangerous than forgetting it entirely.
3. I Even Visualized the Act of Remembering
PrismD's visualization wasn't just a list of nodes. You could right-click a memory, select "trace spreading activation," and watch as the source node lit up, then the first hop, then the second hop — like electric current traveling through a neural network in the dark.
Nodes would also dim in real time. The frontend replicated the backend's exponential decay formula, making "forgetting" visible for the first time.
This confirmed for me that the algorithm wasn't a black box: why a particular memory was recalled, where the energy came from, which hop it decayed at — all observable.
But "explainable" doesn't equal "worth using." A beautiful dashboard can't answer the question of ROI for the architecture.
4. In the End, I Used a Controlled Experiment to Overturn Myself
I didn't just say "the graph is too heavy" based on feeling. I first reimplemented SpreadActivation 1:1 in zero-dependency Python and verified each item:
- Does energy strictly decay as
0.9 × 0.85^hop? - Does
ToolLog → UserBase = 0.05actually block contamination? - Does graph diffusion cover ground truth with fewer tokens?
- After DRIFT, are low-energy nodes correctly pruned by the threshold?
All four checks passed. In other words, graph diffusion was correct.
Then I ran the same real corpus against the same ground truth, pitting three approaches against each other:
| Experiment Arm | Approach |
|---|---|
MD_FULL |
Full Markdown injection as brute-force baseline |
STRUCT_MD |
Structured Markdown + bigram selector, recalling within budget |
LOOM |
PrismD graph diffusion recall |
The winner wasn't the most brain-like LOOM. It was the simplest STRUCT_MD.
Graph diffusion could indeed wake up indirect associations, and it did save tokens compared to full injection. But in a complete engineering context, its marginal recall advantage wasn't enough to offset the maintenance cost of the graph structure, edge weights, cross-cluster matrix, decay state, persistence, LLM-driven organization, and multi-language runtime.
Structured MD was fast enough, transparent enough, easy to reproduce, easy to test, and when something went wrong, you could open the file and check.
So PrismD, after two months of work, was archived. The memory mainline converged to a simpler solution.
This isn't "project failure." It's a complete architecture experiment:
First prove that the complex solution actually works. Then prove it's still not worth it.
Many projects only do the first half — get a demo running and declare victory. Real engineering judgment lives in the second half: is the benefit large enough to justify the entire team paying maintenance cost for it forever?
The path I kept from this experiment:
Pure MD
↓ First: "visible and editable"
Lightweight index and backlinks
↓ Next: "findable"
Source / confidence / time — minimal metadata
↓ Finally: "trustworthy and expirable"
Only introduce vector search or graph relationships when real data proves it necessary
Like planting a tree: let it live first, then prune. Let the trunk grow first, then decide where to graft. Don't install satellite monitoring and city-scale drainage for a seedling on day one.---
VI. The Pitfalls I Stepped In: A Memory System's Worst Enemy Isn't Forgetting — It's Messy Recording
1. Don't Mistake "Machine-Parseable" for "Human-Maintainable"
JSON is great for exchanging structured data. But when memory needs to be read, manually edited, and version-compared over long periods, Markdown is often friendlier. The format serves the scenario — don't canonize any single file type.
2. Don't Let Your Agent Record Everything
Auto-writing on every workflow run quickly turns the memory store into an attic: old delivery boxes, newspapers, a broken fan you can't bear to throw away. The thing you actually need becomes impossible to find.
Auto-memory needs at least a threshold: is it duplicated? Is it stable? Did it come from user confirmation? Will it be useful for future tasks? Otherwise, "growth" is just a taller pile of garbage.
3. Don't Fall for Graph Engines Too Early
Graphs are beautiful, and multi-hop reasoning is seductive. But every new relationship type adds a new state that needs to be explained, updated, and tested. Run the simplest approach first to surface real requirements, then decide where complexity should go.
4. Don't Underestimate the Index
index.md looks like just a table of contents. In reality, it's the foyer of the memory system. If the foyer is clear, the agent knows which door to push. If the foyer is cluttered with junk, even the most luxurious rooms are unreachable.
5. Don't Just Record the Conclusion — Record the Source
"User likes blue" and "the user actively chose blue in three different projects" are not the same kind of memory. Source determines credibility. Time determines whether it's expired. Scope determines whether it can transfer to the next project.
6. Don't Assume the Architecture Is Correct Just Because the Algorithm Is
PrismD's diffusion, decay, and cluster isolation all passed unit tests. But the full system still lost to structured MD. A single gear turning beautifully doesn't mean the whole machine is worth building.
7. Don't Dress Up Sunk Cost as Technical Conviction
The most dangerous thought is: "We've been at this for two months — let's stick with it a little longer." Code doesn't automatically become worth maintaining just because the author can't bear to let go. When the experiment has already answered the question, archiving is more professional than stubbornness.
VII. In Closing: Good Memory Makes an Agent Feel More Like Your Partner Over Time
Back to the beginning: an agent's memory is a web that's constantly being pruned.
RAG solves "where to find material." Graph diffusion solves "how related concepts wake each other up." Pure Markdown solves "how to keep memory transparent." Structured Markdown solves "whether this memory can be trusted."
They aren't four armies fighting each other. They're more like four different tools: call the librarian when the bookshelf is too big, lay out a map when the relationships are too deep, use index cards when you need transparency, add labels when you need reliability.
The final answer PrismD left me isn't "graph diffusion is useless," and it isn't "Markdown is always best." It left me with a simpler principle: Complexity must buy its freedom with real returns.
A truly mature memory system might not be the one that remembers the most, or the one that looks most like a brain. It's the one that knows its limits:
Write when it matters. Hit when it's needed. Let go when it's time.
If you'd rather see how this memory philosophy lands in a real agent — how the index resolves, how context gets injected, how memory participates in the next task — you're welcome to check out the full source code of ResceneAgent on GitHub:
🐙 GitHub: Rescenix/ResceneAgent
If this direction resonates with you, feel free to drop a Star, open an Issue, or browse the code directly. Your feedback will become the starting point of this agent's next "memory."
References & Further Reading
- Alan M. Turing, Computing Machinery and Intelligence, Mind, 1950. Source archive: The Turing Digital Archive, King's College Cambridge
- Tim Berners-Lee, The World Wide Web: A very short personal history: W3C
- John Gruber, Markdown: Syntax: Daring Fireball
- Patrick Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020: Paper page
- Allan M. Collins & Elizabeth F. Loftus, A Spreading-Activation Theory of Semantic Processing, Psychological Review, 1975: DOI: 10.1037/0033-295X.82.6.407
- Jorge Luis Borges, Funes the Memorious, 1942. Cited for its literary insight into how "inability to forget" can hinder abstraction.
- ResceneAgent project source code and documentation: GitHub Repository
- Leon O. Chua, Memristor—The Missing Circuit Element, IEEE Transactions on Circuit Theory, 1971: DOI: 10.1109/TCT.1971.1083337
- Dmitri B. Strukov et al., The Missing Memristor Found, Nature, 2008: DOI: 10.1038/nature06932
- PrismD digital hippocampus engineering retrospective:
prismd-archive/README.md; graph diffusion, decay, and inter-cluster propagation matrix implementation:Prism/internal/memory/graph.go.
Top comments (0)