84 Days, 7,635 Memories, Zero Retrieval Cost: Building a Long-Term Memory System for My AI Agent
A fully local memory system for AI agents. Embedding runs on-device, vectors stay on your disk. The only thing that leaves your machine is the LLM inference prompt. 84 days in production, survived agent crashes, ChromaDB rebuilds, and framework rollbacks — zero memory loss.
Architecture Overview
Before diving into specifics, here's the full system logic:
User Message
│
┌────────────────┼────────────────┐
▼ ▼ ▼
Shallow Recall Deep Pipeline L0 Archive
(Hot Path) (Async, cron) (JSONL text)
│ │ │
│ ┌─────────┼─────────┬──────────┐
│ ▼ ▼ ▼ ▼
│ L1 L2 L3 L4
│ Semantic Scene Persona Knowledge
│ Extraction Blocks Evolution Graph
│ │ │ │ │
│ └─────────┼─────────┴──────────┘
│ │
│ ▼
│ MemPalace (local)
│ ┌───────┼───────┐
│ │ │ │
│ ▼ ▼ ▼
│ drawers closets KG
│ 7,635 2,096 6,077
│ │ │ │
│ ┌────┴───────┴────┐
│ ▼ ▼ ▼
│ Vector BM25 KG Graph
│ (bge-m3) (FTS5) (SQLite)
│ │ │ │
│ └───┬───┴────────┘
│ ▼
│ Fusion Ranking
│ │
└────────┼────────┘
▼
Context Injection → Agent
Two paths, one storage.
The left path is shallow recall — daily conversations take this route. bge-m3 local embedding → ChromaDB vector search, sub-200ms. The right path is the deep pipeline — async cron jobs, L1 semantic extraction calls the LLM API (the only data leaving the machine), L2-L4 run entirely locally.
In practice, 90% of conversations use the shallow path. When the user says something like "recall our discussion about the architecture decision", the system switches to deep retrieval — fusion search across vectors, BM25 keywords, and knowledge graph relations, all deduplicated and ranked before injection.
What I Mean by "Long Memory"
Not just "stored a lot, for a long time." I define long memory by three criteria:
1. Lost data can be recovered.
Agent crashes. System reinstalls. Index corruption. Can the memory survive independently?
On July 9, my ChromaDB index got corrupted. I deleted the entire vector collection, exported all 7,635 drawers from SQLite, and rebuilt the index in 5 minutes. The agent came back online with identical recall precision. Later, an OpenClaw version upgrade broke compatibility and forced a rollback. The L0 JSONL archives and MemPalace database were completely independent of the framework runtime — after rollback and rebuild, every memory was intact.
Migrating to a new machine is just copying two folders. ~/.mempalace/ and the JSONL archive directory. Install dependencies, and the full memory is restored. No re-importing, no re-training. Your memory is a folder, not a proprietary cloud format.
2. Precise cross-day recall, not fuzzy matching.
On April 30, I discussed a project architecture decision with my agent. On July 23 — 84 days later — I asked "what did we decide about that architecture approach?" The agent retrieved the exact discussion, the decision rationale, and the final conclusion. This works because three retrieval lines fire simultaneously: L0 raw text locates the time range, L1 structured memory extracts the facts, L4 knowledge graph adds related entities.
Without long memory, an agent facing this question can only search the last few dozen conversation turns. Anything beyond the context window is invisible. With this system, a discussion from 84 days ago follows the same retrieval path as yesterday's conversation.
3. One sentence triggers deep search. No query writing required.
How does the agent know when to use shallow vs. deep recall? Not by asking the LLM — that's unreliable. The trigger is pattern matching in the system prompt layer: when a user message contains "认真回忆" (recall carefully), "深度检索" (deep search), "还记得吗" (remember?), or similar patterns, the agent switches to the deep retrieval pipeline. Everything else defaults to shallow, keeping daily interactions fast.
When triggered, it's not a single vector search. The pipeline:
- Scans L0 JSONL archives for relevant time ranges (plain text, zero cost)
- Runs bge-m3 semantic search on L1 structured memories
- Enriches results with L4 knowledge graph — find "architecture decision", KG traces back to "project name", "stakeholders", "follow-up decisions"
- Merges and ranks by fusion score (vector similarity × BM25 keyword × KG relation weight × time decay)
- Deduplicates and injects into agent context
To the user, it's one sentence. Deep retrieval latency is under 500ms — embedding and indexing are all local, no network round-trips.
Shallow vs. Deep: The Key Architecture Decision
If every conversation ran the full L0-L4 pipeline — embedding, extraction, summarization, graph building — the latency would be unacceptable. But if we only do fast vector search, cross-day precision recall is impossible.
The solution: split memory into two layers sharing one storage backend.
| Shallow (Hot) | Deep (Cold) | |
|---|---|---|
| Scope | Recent conversations | Full history |
| Path | bge-m3 embed → ChromaDB vector | Full L1→L4 pipeline |
| Latency | <200ms | <500ms |
| Trigger | Automatic (default) | Keyword-triggered |
| Covers | 90% of daily context needs | Cross-day/week precision recall |
This split also means fast disaster recovery. The L0 JSONL archive is plain text, completely independent from the agent runtime. Agent crashes? Archives are fine. Vector index corrupts? Archives are fine. As long as L0 survives, every layer above can be rebuilt from source within a day.
Embedding and Retrieval Sovereignty
This might sound niche, but it matters more than the LLM model itself.
Cloud-based memory services typically package embedding and retrieval as a black box. You send conversations in, the provider decides what embedding model to use and how to rank results. Most English embedding models have poor semantic discrimination for non-English languages. But in cloud services, you can't swap the embedding model.
I did two things:
Embedding layer: Replaced MemPalace's default 384-dim English model with bge-m3 (BAAI, 1024-dim multilingual). For the same query "system architecture design", before the swap: fragments containing those keywords. After: semantically relevant discussions about architecture design. For non-English scenarios, this is a qualitative gap.
Retrieval layer: Fusion ranking instead of pure vector similarity. Three signals combined — vector semantics (bge-m3), keyword matching (SQLite FTS5 with CJK tokenizer), and knowledge graph relation weights. Pure vector search tends to rank temporally distant but semantically similar results too high; fusion ranking corrects this with relation weights and time decay.
Neither of these has anything to do with which LLM you use. They determine the recall quality floor — the best LLM can't answer correctly if it receives the wrong information.
Three Real Problems, Three Reusable Solutions
1. Swapping embedding models bricked ChromaDB
After upgrading to bge-m3 (1024-dim), ChromaDB refused to open the old collection. Collection dimension is immutable after creation — the old metadata was locked at dimension=384. The MCP server saw 384≠1024 and rejected initialization. Search was completely down.
Solution: rebuild, don't patch. Export all drawers → backup (JSON + SQLite) → delete old collection → create new with bge-m3 → re-embed in batches. 7,635 drawers, fully automated in 5-10 minutes. Anyone upgrading from English to multilingual embedding will hit this — the migration script is reusable.
2. OpenClaw upgrade, forced rollback, zero memory loss
An OpenClaw version upgrade caused compatibility issues, forcing a rollback. Configurations were partially overwritten during the process, indexes needed rebuilding. But L0 JSONL archives and the MemPalace database were fully independent of the OpenClaw runtime. Rollback, rebuild — the memory library was untouched.
3. Mac sleep killed all 8 cron jobs
The system relies on 8 launchd scheduled jobs. macOS launchd's StartInterval doesn't catch up on missed schedules after sleep. Mac sleeps for 8 hours — every pipeline comes up stale. Morning health checks show all red — not because anything's broken, but because the timers froze.
Solution: proactive healing, not a different scheduler. A preflight health check runs at the top of every maintenance cycle. It scans all pipeline log timestamps, and for any that exceed their interval threshold, it runs the job immediately and touches the log. Every morning, the first message triggers self-healing — all missed overnight pipelines run automatically.
Data (84 Days)
| Metric | Value |
|---|---|
| Runtime | 84 days (Apr 30 – Jul 23) |
| Structured memories | 7,635 drawers, auto-categorized by project/topic/time |
| Knowledge graph | 6,077 active triples, timeline query support |
| L0 archive | 97 files, 6,136 JSONL entries, 18MB |
| Total storage | 194 MB (drawers + closets + KG + vector index) |
| Retrieval latency | <200ms shallow, <500ms deep (fusion) |
| Retrieval cost | Zero — all local embedding and search |
Same task, with and without long memory:
Agent receives "what did we decide about that architecture approach?"
Without long memory: searches the last few dozen conversation turns. Anything beyond the context window is invisible. Response is either "I couldn't find that information" or worse — hallucinates something plausible but wrong.
With long memory: L0 locates the April 30 conversation → L1 extracts decision points → L4 adds project entity relations → answer includes exact discussion time, decision content, and context.
Cost perspective: Cloud memory services charge per retrieval. This system performed thousands of retrievals over 84 days — cost: zero. Additionally, because the agent has stable memory context injection, users don't need to re-explain background every conversation. LLM prompts are more compact and predictable, leading to higher cache hit rates.
Hardware
Works on both Mac and Windows. No special hardware — the embedding model runs through Ollama, GPU makes it faster but isn't required. Storage and retrieval cost nothing. Migrating to a new device means copying two folders.
Author: CarsonLiu | July 2026 · Based on 84 days of production data
Chinese version: https://juejin.cn/post/7665530652353445894
Top comments (0)