Every AI agent forgets everything when the session ends. Fixing that usually means wiring up a vector database, an embedding pipeline and a retrieval service — a platform, with servers to run and backends to choose. I wanted the other thing: a primitive. npm install, one file on disk, done. The SQLite of agent memory.
So I built rememori: an embedded memory engine in pure TypeScript with zero runtime dependencies. This post is about the three design decisions that turned out to matter — the scoring formula, the entity graph, and an HNSW index that taught me a lesson about benchmarks.
Three verbs
The whole API:
import { Memory } from 'rememori';
import { ollama } from 'rememori/embedders';
const mem = await Memory.open('./agent.mem', {
embedder: ollama('nomic-embed-text'),
});
await mem.remember('User prefers dark mode', { tags: ['prefs'], importance: 0.8 });
const hits = await mem.recall('UI settings?', { limit: 5, halfLifeDays: 90 });
await mem.forget(hits[0].id);
The embedder is an interface, never bundled — local Ollama, any OpenAI-compatible endpoint, or your own (texts) => Float32Array[]. That single decision is what lets the same engine run in Node, Bun, and (with transformers.js) entirely inside a browser tab.
Similarity alone is not recall
Pure cosine similarity is a fine search primitive and a poor memory. Two problems show up immediately when you use it for an agent:
Time doesn't exist. A preference stated yesterday and one stated two years ago rank identically. Human memory doesn't work like that, and agent memory shouldn't either.
Names beat words. Ask "any updates from Giorgio?" and the embedding of that sentence may be nowhere near the embedding of "fixed the flaky deploy pipeline" — even if Giorgio is the one who fixed it. The connection is the entity, not the phrasing.
So recall in rememori scores every candidate as:
score = (cosine + entityBonus) × importance × 0.5^(age / halfLife)
-
entityBonus = min(0.3, 0.1 × shared entities)— memories that mention the same entities as the query get a boost, capped so it never swamps genuine similarity. -
importanceis a 0..1 weight you set at write time. - The decay term is optional and per-query: a half-life of 90 days means a six-month-old memory needs roughly 4× the raw relevance to outrank a fresh one.
The entity term is what I'd defend hardest. At write time, every memory is linked to the named entities it mentions — a bipartite memory↔entity graph. The default extractor is a zero-dependency capitalization heuristic (pluggable if you want an LLM doing it). At recall time, entities are extracted from the query too, and any memory sharing one is guaranteed into the candidate set. A memory with zero textual similarity can still surface because the who matches. In practice this rescues exactly the queries that pure vector search fumbles: people, projects, product names.
One file on disk, structured clone in the browser
Storage is an adapter interface with three real implementations:
-
Node/Bun: an append-only JSONL log. Writes are appends, deletes are tombstone lines, and a compaction pass rewrites the file. Crash-safe by construction,
grep-able by accident, and you can read your agent's memory withless. -
Browser: IndexedDB, selected by path (
idb://agent). A pleasant surprise here: IndexedDB storesFloat32Arraynatively via structured clone — no base64, no serialization layer at all. -
:memory:for tests.
The browser storage is why the demo on rememori.dev works the way it does: transformers.js generates embeddings locally, IndexedDB persists them, and after the model downloads once you can watch the Network tab stay silent while you store and semantically recall memories. Pure TypeScript is the load-bearing constraint — native bindings would have stopped at the browser's edge.
The HNSW index, and the benchmark that lied to me
Exact search over contiguous Float32Arrays is genuinely fast — ~0.5ms per query at 1,000 memories. Most agents never need more. But "it doesn't scale" is a fair objection, so v0.4 added a pure-TS HNSW index: hierarchical navigable small world graph, the same algorithm behind most production vector databases. About 250 lines including a hand-rolled binary heap.
First benchmark run, 10,000 vectors:
recall@10 = 36.8%
Ouch. I spent an hour hunting the bug. Neighbor selection heuristic — textbook. Layer assignment — textbook. Beam search termination — textbook. Then I ran the same index on 32-dimensional vectors: 100% recall at every size.
The bug wasn't in the code. It was in the data. I was benchmarking on uniformly random 384-dimensional vectors, and in high dimensions random points concentrate: every pair is nearly equidistant, there's no neighborhood structure, and no graph-based index can navigate structure that doesn't exist. This is a documented pathology — the hardest datasets in ann-benchmarks are exactly this shape — and it has nothing to do with real workloads. Actual embedding models produce vectors on low-dimensional manifolds; that structure is the whole reason embeddings are useful.
Re-benchmarked with realistic data (384-dim vectors with 24 dimensions of intrinsic structure, queries as perturbations of stored points — i.e., a paraphrased question against a stored memory):
| memories | exact scan | HNSW | recall@10 |
|---|---|---|---|
| 1,000 | 0.5 ms/query | (index off — scan wins) | 100% |
| 10,000 | 4.4 ms/query | 2.3 ms/query | 100% |
| 50,000 | 26 ms/query | 2.8 ms/query | 100% |
Two takeaways I'd pass on:
- If your ANN benchmark uses uniform random high-dim vectors, it's measuring the wrong universe. Generate data with intrinsic structure or use real embeddings.
-
Below ~1,000 items, don't index at all. The exact scan wins on latency, memory and simplicity. rememori's default (
index: 'auto') only builds the graph past that threshold — and tag-filtered recalls always use the exact scan, because approximate + filtered is where correctness quietly dies.
What it deliberately isn't
No multi-user server, no auth, no cloud sync, no document-ingestion pipeline. If you need those, Cognee and Mem0 are good platforms. rememori is for when you want memory inside your process, in five minutes, with nothing to operate.
One more thing that fell out of the design: since the engine is embedded, wrapping it in an MCP server took an afternoon. claude mcp add rememori -- npx -y rememori-mcp gives Claude Code — or Cursor, or any MCP client — persistent semantic memory across sessions, stored in a local file, embedded via local Ollama.
rememori is MIT-licensed and early (v0.4). The name is Esperanto for "to remember" — a language built to run anywhere, which felt right. Site and in-browser demo: rememori.dev. Feedback I'm most interested in: the recall formula, the storage format, and what you'd need before trusting it in a real agent.
Top comments (6)
The formula's clean, but one thing stands out for real-agent use: importance is fixed at write time and nothing reinforces a memory when it actually gets recalled. So a fact you hit every day decays at the same rate as one you never touch again, which is the opposite of how memory usually hardens. Did you consider bumping importance (or resetting age) on recall, or does that risk a feedback loop where an early wrong memory entrenches itself just because it keeps surfacing?
Great point, and you've put your finger on exactly what v0.6 is about.
You're right that importance is write-time-only today. The reason recall() doesn't auto-reinforce is deliberate though: recall is currently a pure read — no side effects, idempotent, safe to call speculatively. Auto-bumping on recall would break that, and more importantly it conflates surfacing with being useful, which is precisely the feedback loop you describe: an early wrong memory keeps ranking, keeps getting "reinforced", entrenches.
The design I'm converging on for v0.6: an explicit reinforce(id) — the caller signals "this memory actually made it into the answer", not "this memory was a candidate". That breaks the loop because the reinforcement signal comes from outside the ranking system. Mechanically I'd rather reset the decay anchor (decay from last-useful-recall instead of creation) than inflate importance, and damp it logarithmically so a daily-hit memory hardens but can't run away.
The failure mode that remains is a wrong but genuinely used memory — but that's not a ranking bug anymore, that's the agent needing forget(). Curious if you'd model it differently.
Update: shipped. reinforce(id) landed in v0.6.0, pretty much as described above — decay anchor moves to the last reinforcement (a memory you keep using stops ageing), plus a log-damped bonus capped at +0.15 so it hardens but can't run away. recall() stays a pure read; the reinforcement signal has to come from the caller, i.e. from outside the ranking system — that's the part that breaks the entrenchment loop.
github.com/GiorgioDotcom/rememori — thanks for the push, this was the best kind of comment.
Love that you shipped it instead of just theorizing, and the +0.15 cap is the right instinct. One hinge I'd still poke at: reinforce(id) is only as clean as whoever asserts the memory "made it into the answer." If that caller is the agent saying it was useful, you've quietly rebuilt the entrenchment loop one level up, with self-report deciding what hardens. It gets trustworthy when reinforcement fires on verifiable use: the memory was quoted in the output or drove a tool call with matching args, not "the model felt it helped." On the forget() gap you flagged: an explicit call needs someone to notice, which rarely happens on time. Contradiction is the trigger that scales, when a new memory conflicts with an older one that's still ranking, demote on the conflict instead of waiting. That also splits your last failure mode: an unused memory just decays, a used-and-wrong one takes a negative signal from the bad outcome, so "never recalled" and "recalled and harmful" stop sharing the same path down.
Both points land, and the second one has a fun twist buried in it.
On self-report: agreed, "the model felt it helped" just rebuilds the loop one level up. The direction I'm taking is making the verifiable path the easy path: a helper that takes the recalled hits plus the final output and reinforces only hits with substantial textual overlap (quoted spans, or tool-call args that match the memory's content). Evidence-gated, not vibe-gated. The honest limit: paraphrased use slips through n-gram matching, and I'd rather under-reinforce than trust embedding similarity between memory and answer — that's the soft version of the same loop.
On contradiction: fully agree it's the trigger that scales, but here's the twist — embeddings can't detect contradictions, they embrace them. "The deploy needs Node 20" and "the deploy needs Node 18" are nearly identical vectors. So the engine's honest job is collision detection, not contradiction detection: at write time, surface "this new memory is suspiciously close to these existing ones", hand the pair to the caller (who has an LLM and can judge duplicate vs update vs conflict), and provide the primitive to act on the verdict — a demote that works as negative reinforcement, so a used-and-wrong memory takes an explicit hit instead of waiting for decay. Your split of the failure modes ("never recalled" vs "recalled and harmful") is exactly the design test I'll hold that against.
This is turning into the v0.7 spec, honestly. Thanks for keeping the bar high.
Round two shipped — v0.7.0, and your comment basically wrote the spec:
demote(id) — symmetric negative feedback, log-damped, capped at −0.15. Your split held up as the design test: an unused memory just decays, a used-and-wrong one takes an explicit hit. Two different paths down.
reinforceFromOutput(hits, output) — reinforcement is now evidence-gated: a memory only hardens if its content verifiably appears in the final answer (a quoted token run, or high distinct-token containment). Deliberately dumb text matching, no embeddings — trusting the model's self-report rebuilds the loop one level up, and embedding similarity between memory and answer is the soft version of the same mistake.
collisions(id) — your contradiction trigger, with an honest division of labor: embeddings can't detect contradictions, they embrace them ("needs Node 20" vs "needs Node 18" embed nearly identically), so the engine surfaces proximity at write time and the caller's LLM adjudicates duplicate vs update vs conflict, then demotes or forgets the loser.
A war story you'll appreciate: a fresh-context review pass caught that my first evidence tokenizer ignored 1–2 digit numbers — so an answer saying "Node 18" counted as evidence for a memory saying "Node 20". Your exact example, reinforcing the losing side. Numbers now veto both evidence paths: a quote that changes the number is not a quote — the number is the payload.
Remaining open edge: paraphrased use slips past text matching, so reinforcement under-fires by design. I'd rather have that than false hardening — but if you have a verifiable signal for paraphrase-level use that doesn't reintroduce self-report, I'm all ears. You're two for two on writing my roadmap; the tools are live in rememori 0.7.0 / rememori-mcp 0.3.0 if you want to poke at them.