Every Agent ships with memory now. It's just assumed.
So let me ask: what are you actually trying to solve by adding it?
Is the core of recall accuracy, or usefulness?
Where does your memory live, who controls it, can you take it with you?
Is it just a database that gets queried, or does it participate in understanding the user?
Is what the user said what the user actually wants?
Should a user's memory belong to the user?
Those questions all point me one direction: build a local, vector-free memory.
Local, because memory is an asset — and assets stay in your hands.
No vectors, because I want useful, not accurate.
Why It's an Asset
Everything you talk about with an AI — your preferences, your habits, the mistakes you've made, the context you've explained three times over — that's an asset.
Lose a cache, who cares. Lose an asset, that's a real loss.
So where does it live? Most memory systems keep it on someone else's server. Your conversations get extracted into facts, vectorized, pushed to the cloud. You can't see it, can't change it, switch platforms and it's gone. And anything involving work or client information — once it leaves your machine, you've lost control of it.
So the first thing a memory system should figure out isn't how to recall better. It's who owns these memories, who holds them, and whether you can take them with you.
1. Usefulness
Everyone's racing on recall precision right now. RAG, vector stores, rerankers — each generation better than the last.
But is the core of recall accuracy, or usefulness?
No matter how precise, it's still search. You ask something, the system dumps everything "relevant" into context and burns a pile of tokens.
Say you're chatting and ask, "hey, what happened with that thing?"
What you want is: "That review last Wednesday, right? Option A went through."
What a recall-optimized system does is dump the entire event record for that review. The cause, the process, the conclusion, who was there, how you felt, the follow-up discussion. All of it.
It's accurate. 100% relevant. But do you actually need it?
Think about how people remember.
It's perception-driven. First you sense what the other person needs, then you decide which layer of memory to pull. When you recall something, what surfaces is the outline, not the full text. Details only come out when the conversation goes deeper.
So: three layers. Shallow gives a cognitive summary. Medium gives an overview. Deep gives the full content.
People don't retrieve then answer. They perceive, then respond.
And I don't use vectors.
There's no cosine similarity in your head. Recall happens through features — a sound, a face, a smell, and then "oh, right."
Feature tags are explainable, editable, lightweight. You know why something was recalled, and you can fix it directly. When vectors go wrong you tune parameters and hope. When tags go wrong you can see exactly which two got connected.
But no vectors doesn't mean it's simpler.
Tags are alive. The same idea, and the LLM outputs "tag pool," "tag pool management," "tag library" — three different names. What do you do?
My first instinct was to merge them. That was wrong.
The goal of a tag pool isn't cleanliness. It's full recall.
Merging means deleting an entry point. The next time the user says exactly the word you deleted, that memory is unreachable. So I switched to relational coexistence — don't eliminate tags, link them so they can find each other.
And that's just one problem.
Generic words like "issue" or "handle" only pollute recall, so they need to retire. But when the filter was only in the database layer and not the in-memory cache, they were still being recalled in the same process. Deduplication was worse: retired tags still occupy their name and ID, so if the dedup query filters them out, the system thinks "this doesn't exist," creates a new one, and hits a UNIQUE constraint violation.
All of that I ran into myself. A vector approach won't hit these, but it hits its own set: black box, not editable, can't tune it.
Nothing's free. Just different tradeoffs.
2. Security
This is the one I won't compromise on.
All data stays local. No cloud dependency. Just SQLite and JSON files on your own disk. No embedding service, no sending conversations to a third party, works offline.
Data doesn't leave. Work content, client information, personal habits — not a single line leaves your machine.
White box. Memory is plain JSON. Open it and see exactly what it recorded about you.
Explainable. No vectors, so every recall can explain itself: which tag matched, how it spread to related tags, why this card ranked higher.
Lightweight. Only SQLite and System.Text.Json. No GPU.
Your memory, your custody.
3. Asset Growth
After each conversation, the C-line settles things asynchronously: records the event, writes the overview, completes tags and scene, corrects preferences. The longer you use it, the richer the pool, the more precise the injection.
More importantly, it understands you — not just the conversation content. Your preferences, your habits.
So what exactly is the "digital twin" part?
| Layer | Capability | Example |
|---|---|---|
| One | Remember facts | "He hates long replies" |
| Two | Understand preferences | "Conclusions at work, can ramble when chatting" |
| Three | Read unspoken intent | "He's asking if the plan works, but he's actually hesitating" |
The first two are remembering. The third is what makes it like him.
What users say and what users want often aren't the same thing. "What happened with that thing" might mean "I don't remember the details, just give me a summary." "Does this plan work" might mean "I'm on the fence, give me a verdict."
Only memory can fill that gap. With memory, it knows you said last time "don't give me a list of options, just tell me the answer" — so this time it gives you the answer.
That's why memory shouldn't just be a component that gets queried. It participates in understanding the user.
And the twin is portable. Open formats, so switching models or machines — your assets come with you.
In the End
| My Choice | |
|---|---|
| Memory unit | Feature tags, not fact triples |
| Retrieval | Feature matching + relation spread, not vector similarity |
| Injection | Perception-driven, shallow/medium/deep, not dump-everything |
| Role | Participates in understanding the user, not just a database |
| Storage | Local SQLite + JSON, not cloud |
| Goal | Useful, not accurate |
Recall accuracy still matters. It's the foundation. I just think two layers get discussed far too little: whether to inject at all, how much, and at what granularity — and who these memories belong to, who holds them.
Memory is ultimately the core asset humans accumulate.
Open source: VinsonWild/Wangdefa.Memory
Top comments (0)