How I Built a Local Memory Network for My AI Agents

Your AI agent forgets everything between sessions. So it re-derives the same
decisions, and you pay for it every time.

I got tired of paying. Tired of re-explaining the same project every morning,
watching the agent rediscover a gotcha I hit three weeks ago, re-litigating a
decision I already made.

So I stopped relying on the agent's memory and built the memory into the
projects themselves.

I've been running this system across everything I build for months now: a
SaaS, a couple of games, personal tools. This post is the full internals. How
the files are structured, how they link to each other, and how the whole thing
became searchable in about 0.2 seconds without a single byte leaving my
machine.

The whole idea in one sentence

The repository itself is the memory.

Every durable fact gets written to exactly one file, committed where the work
happens. Decisions, conventions, gotchas, what I'm working on right now. A
fresh agent session doesn't recall any of it. It reads it.

That sounds obvious. Maybe it is. But the difference between "we have some
docs" and "the repo is the memory" comes down to three things most doc folders
never get:

• A map, so an agent can find the one file it needs without scanning the tree
• Links, so facts have one home and everything else points at it
• Search, for when you don't know which file to open

Let me walk through each layer.

The part I didn't plan

I built this to stop coding agents from forgetting technical context. But once
the network held decisions, current work, past mistakes, meeting notes, and my
own writing preferences, it turned out to be useful outside the code.

It now drafts my documentation, release notes, blog posts, and social posts
from the actual history of the work. I review, correct, and approve
everything, but I no longer have to reconstruct what happened before I can
explain it. This post was drafted from the same network it describes, and the
site it lives on is updated through that system too.

The shape: three tiers

Not everything belongs in a repo. A convention about how I write tests belongs
in that project. A rule about how I write emails belongs everywhere. My own
working-style preferences belong to me, not to the codebase.

So the network has three tiers.

Fig 1. One home per fact. A top-level index file maps all the networks so any session can locate the right one.

The routing rule is the load-bearing part. The first time a rule shows up
that's wider than the current project, it moves up a tier immediately, in the
same session. Otherwise you end up with four copies that slowly drift apart,
and then you can't trust any of them.

Inside a repo network

Here's the anatomy of a single repo's memory, using my biggest project as the
example.

Fig 2. The read path of a fresh agent session. The always-loaded file stays lean on purpose; everything heavy is lazy-loaded through the map.

The always-loaded manual holds only the rules that prevent damage. Everything
else is behind the map, and the map entry for each file includes one line
saying when to read it. That's what keeps the token cost sane. The agent
doesn't front-load forty documents. It locates, then opens one.

That sounds wrong if you're used to "more context is better." But every file
the agent loads costs tokens, and most of them wouldn't have changed the
answer.

How it all links together

Three kinds of links hold the network together, and none of them are clever.

1. Wikilinks between notes

Every memory note is one fact in one file, with a little frontmatter and a
slug. Other notes reference it with [[that-slug]]. A link to a note that
doesn't exist yet is allowed on purpose. It's a marker that something is worth
writing, not an error.

---
name: verify-before-you-report
description: "Never claim a result you didn't read from the tool this turn"
type: feedback
---

Confirm the exit code, re-run the check on the committed tree, then say it.
Related: [[done-means-verified]], [[never-invent-a-sha]]

2. Index files as hubs

Every tier has a small index: one line per note with a pointer. The indexes
are what get loaded into context; the notes are what get opened on demand.
Same shape at every level, from a single repo's context map up to the
top-level file that maps all the networks on the machine.

3. A health check that keeps the graph honest

Links rot. So a script runs at session start and reports three numbers:
dangling links (a [[slug]] pointing at nothing), orphans (notes no index
points to), and missing file links. This morning it reported 0 dangling and 8
orphans. Those orphans are real debt, but now I can see it instead of finding
out later.

Fig 3. I also generate an actual interactive version of this from the real files: a small script walks the folder, parses the wikilinks, and emits a self-contained force-graph HTML page. Zero dependencies, nothing hosted.

The pattern underneath all three: one home, many pointers. A fact lives in
one file. Everything else links to it and never copies it. Two copies of the
same fact will disagree eventually, and then your memory is lying to you.

How it's searchable (without sending anything anywhere)

Links work when you know roughly where something lives. Search is for when
you don't.

The search layer is one SQLite file, built by a small indexer that walks the
whole network: wide memory, every project's notes, and the docs folders of the
repos I care about. It chunks each file into pieces of about 220 words,
respecting markdown sections, and indexes every chunk two ways:

• Keyword (BM25 via SQLite's FTS5). Catches exact rare tokens: an env var name, a commit SHA, a number like 0.62. Embeddings genuinely miss these.
• Semantic (local embeddings via Ollama). Catches paraphrase, so a query about "pricing" finds the note that says "monetization." Nothing leaves the machine, which matters because this corpus holds financials and unreleased work.

At query time, both searches run and the results get merged by Reciprocal
Rank Fusion. Not by mixing scores. A BM25 score and a cosine similarity
aren't on the same scale, so you fuse by rank instead, and it needs no tuning.
Answers come back in about 0.2 seconds over roughly eight thousand chunks.

Fig 4. If the local embedding model isn't running, it falls back to keyword-only instead of failing. The search has to always answer something, or I'd stop reaching for it.

Two design choices here matter more than the retrieval math.

The index is disposable. Markdown is the source of truth. The database is
derived from it, and I can delete it and rebuild from scratch in about a
minute. There's no product sitting between me and my own notes, and nothing to
migrate away from later. The notes are still just files on my machine.

Indexing is incremental by content hash. A run with no edits is a no-op
that takes milliseconds. That's what makes it cheap enough to just always be
current.

The agent gets this as a one-line query tool, and the standing rule is to
orient first: before any real task, ask the network what's in flight on that
topic. The answer is either "here's the context" or "nothing relevant," and
both are useful.

The habits that keep it alive

Honestly, this is the part that decides whether the whole thing works. The
files and the search are just plumbing. What actually kills a system like this
is nobody writing anything down.

Three habits, and the mechanisms that enforce them:

Capture in the same pass. When a decision gets made or a gotcha gets hit,
it goes to its one home in that same session. Not a follow-up chore, part of
"done." The agent does the writing; my job is to not skip it.

Hooks that surface memory instead of trusting recall. A session-start hook
prints the open items and the current focus before any work begins. Another
hook fires right before a long conversation gets compacted and re-surfaces the
capture rules, because compaction is exactly the moment a learning would
otherwise be summarized away.

One-word verbs for the expensive moments. Ending a session well used to
mean writing a handoff by hand, so I mostly didn't. The data made that
embarrassing to look at:

Fig 5. 171 session endings, 4 handoffs. I was stretching sessions instead of ending them, because ending them well was work.

The fix was to make the whole close-out cost one word. I type handoff and
the agent writes the summary, files the open questions, sweeps for uncaptured
gotchas, and stores it where the next session's hook will surface it. I type
pick up and the next session starts exactly where the last one stopped. Same
idea for idea (log a passing thought without derailing the session) and
queue (one canonical answer to "what's first today," computed by a script
instead of vibes).

One word removes the friction. The friction was the whole problem.

What this is not

It isn't a hosted product or a service I'm asking you to buy. It's also not
magic memory for the AI. The agent still forgets everything between sessions. The system
just makes forgetting cheap, because everything worth knowing is written down
where the next session will look.

And the AI doesn't get this right on its own. It captures the wrong things,
files duplicates, and lets links rot, unless the rules are written down and
enforced by the same network it maintains. The judgment stays with me. The
system just means I only have to make each call once.

If you want to start

You don't need the search index or the graph or the hooks on day one. I didn't
have them either. The minimum that pays for itself immediately:

• One short always-loaded file: what the project is, how to run it, the rules that matter
• docs/now.md: what's active right now, a few dated lines
• docs/decisions.md: one dated entry per real decision, with the why
• docs/notes.md: gotchas worth not hitting twice
• The habit: write it down in the same session it happens

Everything else in this post grew out of that seed, one annoyance at a time.
The linking showed up when facts started needing two homes. The search showed
up when I couldn't remember which file held a decision. The verbs showed up
when I noticed I was avoiding my own system because it cost too much to use.

The public repo, Build With a Memory,
includes the seed version: a single prompt you paste into an agent to install
the basic structure in an existing project. The tools from this post are in
there too, under tools/:
the search indexer, the query tool, the graph generator, and the link-health
check, generalized to run on any folder of notes.

Writing it down costs less than re-explaining it.

What are you building into your repos?

I'd honestly love to hear how other people are handling this. If you've got a
memory setup for your agents, even a messy one, what's the piece that actually
gets used? And what died from neglect?

Credits

The network is my own build, but the way of working it grew out of is not all mine. A talk by Matt Pocock flipped committed project memory from something I toyed with into how I actually work. The method repo keeps the full credit record in CREDITS.md. If this post uses an idea I have not credited there, tell me and I will fix it.