I built a local-first memory service so my coding agent stops forgetting between sessions

My coding agent was very good at solving a problem and very bad at remembering that it had. Every session started clean. Which port the local service listens on, why we went with one crate instead of another, the fix for a race we had already chased twice: I typed all of it again, in a slightly different order, most days.

Each harness has some answer to this. pi has its own store, Claude Code has another, Codex keeps transcripts. None of them hand anything to the next one, so the memory lives wherever you happened to be working that week.

I wanted one place to keep it, on my machine, that any of them could ask. That turned into memnest.

One service, several front doors

memnest is a single Rust binary listening on 127.0.0.1:3111. One address serves both an HTTP API and a Streamable HTTP MCP endpoint.

http://127.0.0.1:3111        HTTP API
http://127.0.0.1:3111/mcp    MCP endpoint

Each harness exposes different extension points, so the wiring differs while the store and the tool contract stay identical:

Harness	Prompt-time recall	Memory tools	Transcript capture
pi	Autocontext, from the extension	Registered by the extension	memnest watch
Claude Code	memnest hook on UserPromptSubmit	MCP	memnest watch
Codex	memnest hook on UserPromptSubmit	MCP	memnest watch
Other MCP clients	Depends on the client	MCP	Not applicable

Point any MCP client at it and you are done:

{
  "mcpServers": {
    "memnest": { "url": "http://127.0.0.1:3111/mcp" }
  }
}

Streamable HTTP is the recommended transport because every client then shares one server and one data directory. stdio works, but only when that single process owns the store. A second writer on the same data directory is rejected rather than allowed to race the indexes.

No LLM anywhere in the loop

Nothing in memnest calls a model API. Embeddings run locally with intfloat/multilingual-e5-base, and there is no summarization step between what you saved and what comes back.

That was a deliberate choice, and it cost something. LLM summarization would compress a long transcript into something denser and probably rank better. It would also mean that reading my own notes depends on someone else's uptime, someone else's pricing, and a paraphrase I never reviewed. A memory that quietly rewrites what you told it is worse than a memory that occasionally returns too much.

So there are two kinds of records, kept apart. Things you save on purpose (decisions, preferences, corrections) and redacted conversation text, kept verbatim. Both are searchable, neither is rewritten.

SQLite is the truth, the indexes are opinions

The store is SQLite. Next to it sit a Tantivy BM25 index and an HNSW vector index, and both are derived state that can be thrown away and rebuilt.

That distinction is what makes the write path recoverable. A write inserts the record and an index job in the same transaction, then builds both indexes, then clears the job. If the process dies halfway, the job is still there at startup and the write is replayed. The failure mode is a few seconds of extra work on boot, not a memory that exists in the database and cannot be found by search.

Why two indexes and not one

This is the part I would argue about with someone.

BM25 finds an exact token. Ask it for 3111 or a crate name and it lands on the right record. Ask it for the same idea in different words and it shrugs.

Vector similarity does the opposite. It handles the paraphrase and drifts away from the literal string you actually typed, which is exactly the case where you knew the answer and just wanted it back.

You cannot tell which one you need until the query arrives, and by then the write is long done. So a write pays for both, and a read merges the two rankings with reciprocal rank fusion at k=60, then reranks with MMR at lambda=0.5 so five near-identical memories do not fill the whole result.

Scope narrows before any of that runs: the current directory's workspace plus a shared playbook scope for rules that hold everywhere. Project memory does not leak sideways.

What I measured, including what I threw away

The benchmarks doc has a public fixture of 22 documents and 46 hand-labelled Korean queries, 41 with a relevant document and 5 deliberately with none.

Current hybrid search gets recall@1 of 0.976 on it. Two ideas that looked good did not ship.

A CJK 2-to-3-character ngram tokenizer took partial-term lexical recall from 0.167 to 1.000 on an isolated 5,000-document probe. On the real 46-query fixture it changed nothing, at 2.94 times the index size and roughly 50 times the lexical query cost. Every language pays that, and the measured gain on representative queries was zero, so it stays off behind a flag.

A jina-reranker-v2-base-multilingual cross-encoder took Precision@1 from 0.200 to 1.000 on five known rank errors. It also wanted 1.1 GB of model cache and 35 seconds of warm initialization for a service meant to answer a prompt hook in milliseconds. Also off.

Both are written down with their numbers rather than quietly dropped, because "we tried it and here is what it cost" is the part I always want from someone else's repo and rarely get.

The honest weak spot is candidate rejection. Recall@1 is 0.976, but no-result accuracy is 0.200: four of the five queries that should return nothing still return a low-confidence memory. Good retrieval and bad abstention is a real failure mode for an agent, because a confidently irrelevant memory in the prompt is worse than an empty one. That is what I am working on next.

Secrets do not go in the searchable store

Credentials live in an AES-256-GCM vault behind separate tools, not in anything the search path can reach. Credential-shaped text is redacted on the way in, so a token pasted into a conversation does not quietly become a searchable record.

Install

Linux x86_64 and aarch64 can install a release without a Rust toolchain. The script verifies the archive checksum, installs the binary, registers a user systemd service, and checks its health.

curl -fsSL https://raw.githubusercontent.com/Blue-B/memnest/main/core/scripts/install.sh \
  -o /tmp/memnest-install.sh
bash /tmp/memnest-install.sh --user

Read the script before you run it. Windows and WSL have their own scripts in the same directory, and building from source needs Git and a 2024-edition Rust toolchain.

Starting the service downloads nothing. The embedding model arrives on the first operation that needs it, so the first write or first search is slower than the rest. memnest --warmup-embedding pays that up front.

MIT licensed. The repo is at github.com/Blue-B/memnest, and I would rather hear where the design is wrong than where it is fine.