Retrieval isn't memory: building a reasoning layer for AI

Aleksandar Kovacevic — Mon, 06 Jul 2026 12:56:34 +0000

Anyone who codes with an AI agent knows the feeling. Inside a session it's sharp — it follows your patterns, respects your decisions, fixes the bug. Close the terminal, open it tomorrow, and it's a stranger. It re-suggests the pattern you rejected. It re-introduces the bug you already fixed.

The usual fix is a memory tool — mem0, Zep, Letta, a hand-maintained CLAUDE.md. We tried them and hit the same wall.

Retrieval is not the same as remembering

Every memory tool does the same three things: store text, embed it, find it again. Store → search → return chunks. That's retrieval, and it's useful — but it's passive. It waits to be asked, then hands back the closest matching text.

A coding agent rarely asks. It acts. By the time you'd want it to recall that this repo validates at the boundary, it's already written the code the wrong way. Retrieval that only fires on demand is a library nobody walks into.

So the question wasn't "how do we store more?" It was "how do we get the right guidance in front of the model before the mistake?" That reframing is the whole product.

Three layers over one database

Memory. One SQLite database is both source of truth and vector store, via sqlite-vec — no separate vector DB, no sync job keeping two systems honest. Work is recorded non-intrusively by hooks in the agent. Retrieval is hybrid: vector similarity plus BM25 full-text, then a cross-encoder rerank. This layer is table stakes — roughly what the other tools do.

Angel (reasoning). A persistent process that thinks over what's stored instead of waiting to be queried. It extracts patterns and decides when one is worth surfacing. Before a pattern is promoted, multiple independent passes have to converge on it — if only one run noticed it, it's noise; if several land on the same thing, it's real. That check keeps single-session bias out of long-term memory.

Whisper (behavior). The part worth its own section.

Whispers: steering the model at inference time

Static prompts don't scale. If you've maintained a CLAUDE.md, you know the failure mode — it grows, every rule is always "on," and by the third session the model half-ignores the whole wall.

So we inverted it. The static prompt shrinks to identity essentials. Everything situational — a rule, a lesson, a convention — becomes a whisper: guidance tagged with a trigger condition, dormant until it's relevant. At session start, and on any turn whose shape calls for it, a lightweight detector asks "is this a moment where remembered guidance helps?" If yes, the whisper is injected into context before the model responds.

The model then behaves as if it had internalized the rule — with no weight update. It's the closest analog to fine-tuning without access to the weights: steering through just-in-time context instead of gradient descent. Retrieval makes the model know more; whispers make it behave differently. A database versus a tutor.

The honest part: misfires

Just-in-time injection has an obvious failure mode — firing when it shouldn't. And the costs aren't symmetric: a correct whisper earns a little trust, a wrong one that interrupts your flow costs a lot, because people remember the annoying one.

So the bar isn't "never misfire" — chasing that makes the system too timid to be useful. The bar is: a wrong whisper has to be cheap to ignore. Non-intrusive, easy to dismiss, never blocking. Get that right and thousands of quiet correct nudges beat the occasional shrug.

Where this is

Code stays in your trust boundary — generation runs through your own local agent with your own credentials, not a third-party box in the middle. The reasoning and whisper orchestration is the part that's ours.

The engine runs daily across real projects and benchmarks competitively on long-memory. What we're building now is the boring, necessary part: making a stranger's first session good.

If you've fought agent memory — rolled your own, wrestled a .cursorrules file into the ground — I'd like to hear how you approached it, especially the misfire problem. I don't think anyone has fully solved that one yet.

Sonn is at sonn.dev.

DEV Community: Aleksandar Kovacevic

Retrieval isn't memory: building a reasoning layer for AI

Whispers: steering the model at inference time

The honest part: misfires

Where this is