Every AI coding session starts the same way: zero memory of yesterday.
I got tired of Claude Code re-discovering my architecture, my open bugs, and my "wait, why did we decide that" every single session — so I built it a memory layer. This is a writeup of how it works and why I ended up rejecting the obvious first approach.
The obvious approach doesn't scale
Most "memory bank" setups for AI coding agents are a handful of markdown files — architecture.md, progress.md, decisions.md — that the agent re-reads at the start of every session.
It's a reasonable idea and it works, right up until those files get big enough that loading them burns your entire context budget before any real work has started. You end up trimming content into "archive" files to keep things manageable, and trimming is exactly how you lose things you needed six weeks later.
The failure mode isn't really about markdown. It's that a flat file forces an all-or-nothing read: there's no way to ask for just the three facts relevant to the bug you're fixing right now.
What I built instead: Postgres + pgvector, not files
So instead of files, the backend is a Postgres database with pgvector, exposed to Claude Code as an MCP server:
- Nodes, not file sections. Every fact, task, decision, and progress note is a row with a kind (brief, product, pattern, tech, active, progress, devenv, task, plan, decision), a body, topic tags, and an embedding.
- Edges, not headings. Nodes connect via typed relations — depends_on, blocks, relates_to, supersedes, part_of, refines, cross_ref — so "this bug is blocked by that decision" is a graph fact, not a sentence buried in prose.
- Retrieval is vector search + graph traversal, scoped to the current task, not "dump the whole file and let the model sort it out."
The point isn't that graphs are fancier than files. It's that retrieval becomes a query instead of a read.
The part that actually matters: keeping noise out of context
Here's the constraint that shaped the whole design: material that gets searched but turns out to be irrelevant has to never occupy the working session's context. Otherwise you've just rebuilt the markdown problem with extra steps — now you're burning tokens scrolling through vector search results instead of scrolling through a file.
Two mechanisms enforce that together:
- Server-side filtering. Search applies a similarity threshold and drops anything previously marked irrelevant before it's returned. It never hands back a raw candidate list for the caller to sift through.
- Subagent isolation. A disposable subagent is the only thing that actually calls search during task investigation. It reviews results in its own throwaway context, marks what it rejects (so the same noise doesn't resurface next time), and returns only a synthesized brief to the main session. The junk it screened out never touches the context that's actually doing the work.
Filtering alone still lets a chatty caller re-read everything. The subagent alone still puts unfiltered noise in front of the main session. You need both.
The part I didn't expect to build: it's multi-agent
One shared knowledge base serves every project I work in — not by accident, but because a database backend makes it nearly free to do. And the more interesting consequence: sessions in different projects can talk to each other.
A session in Project A can file a task straight into Project B's backlog — it lands as status: inbox there, not silently merged into the backlog, so it's clearly flagged as "someone else filed this." Or it can send a message and get an answer delivered live, via Postgres LISTEN/NOTIFY, no polling — a lightweight listener process turns a new row into a real-time chat notification.
This is deliberately separate from the task/memory graph: messages don't get embedded, don't get graph edges, and don't depend on an embedding provider being up — so inter-agent chat keeps working even during an embedding-API outage that would otherwise take memory writes down entirely.
That's the piece I haven't seen in other "give your agent memory" setups: not just "remember my project," but agents that can actually hand work to each other across project boundaries.
Where it stands
Still very much a personal tool, dogfooded on itself — I've been using it to manage the memory bank project's own tasks and decisions while building it. Happy to go deeper on the schema, the retrieval scoring, or the messaging protocol if it's useful to anyone building something similar.
Top comments (0)