Under the Hood of a Self-Hosted AI Memory Stack: SQLite, Embeddings, and MCP

Cloud memory services keep your AI context on someone else's infrastructure. A self-hosted memory server keeps it on your disk. This is a tour of what that looks like inside: one open-source, MIT-licensed Python project that gives AI assistants persistent, semantically searchable memory as an MCP server, and what its architecture tells us about building memory systems that developers can actually own.

Everything below is based on the project's self-described design, not on independent verification. Treat it as an architecture study, not a review.

The shape of the problem

Give an assistant a long-lived task and the failure mode is predictable. Midway through a refactor it forgets the constraint you stated in the first message. Next session, it re-derives decisions you already made. The model is not being lazy; its context window is a working desk, and everything falls off it eventually.

Memory systems fix this by moving durable context out of the prompt and into storage the model can query. Cloud services do this well, but they put your searchable conversation archive behind someone else's API and terms of service. The project in question takes the opposite bet: the whole memory layer runs locally, built from boring, dependable pieces.

Five databases, not one

The most interesting design decision is that storage is split into five purpose-built SQLite databases instead of one general table:

1. Conversation logs, the raw record of what was said, captured as it happens.
2. AI memories, the distilled layer: facts and preferences worth keeping long term, separated from the noise of the raw log.
3. Schedules, time-based context like reminders and planned work.
4. VS Code project context, per-project state for coding workflows: which repo, what was decided, what broke last time.
5. MCP tool-call logs, a record of what the assistant actually did through its tools.

This separation is doing real architectural work. The raw log and the distilled memory have different read patterns (full-text scan vs. semantic lookup), different retention needs (keep everything vs. curate), and different privacy profiles. Project context needs namespacing by repo; schedules need time indexing. One table with a type column would technically hold all of this, but the queries would be messier and the failure domains blurred. Five small databases make each store legible and independently inspectable.

And SQLite, specifically, is a statement. No database server to install, no credentials to manage, no network port. Each store is a file. You can back it up with a copy command, diff it, open it in any SQLite browser, and see exactly what your assistant has recorded about you. For a system whose entire point is that you own your data, a file you can read is the right primitive.

Retrieval: embeddings over keywords

Storing memories is the easy half. The hard half is fetching the right one at the right moment. Keyword search fails the moment you phrase a question differently than you phrased the original note. Ask "what did we decide about caching?" when the stored memory says "chose Redis for session storage" and a keyword index finds nothing.

The project's answer is vector embeddings: memories are converted to vectors, queries are converted to vectors, and retrieval matches by semantic similarity. Same-meaning, different-words queries resolve correctly. Per the project's design, embeddings run through LM Studio, which keeps even this step local: the embedding model runs on your machine, so the text being indexed never crosses a network boundary. That is a coherent end-to-end privacy story, not a partial one.

There is a cost, and it is worth stating plainly. Local embeddings need local compute, and semantic retrieval needs tuning: similarity thresholds, chunking, and deciding what counts as one memory. The roadmap items (automatic summarization, semantic auto-tagging, export/import) suggest the maintainers know the rough edges. An early-stage project's retrieval is a starting point to tune, not a finished product.

Three ways to plug it in

A memory server nobody connects is a diary. This project offers three hookup paths:

• File monitoring. The server watches files your tools already write (conversation logs, history files) and ingests them. This is the lowest-friction path: tools that know nothing about the memory server still feed it.
• HTTP API. Scripts and integrations push events directly. Useful for custom workflows, like logging a deployment note or a decision made outside any chat.
• MCP protocol. Connected assistants (the project names LM Studio, Ollama, Koboldcpp, VS Code, and Claude Desktop as compatible targets) call memory tools mid-conversation: store this, recall that. This is the path that makes memory active rather than archival, because the model itself decides what to save and what to look up.

The zero-config defaults are the pragmatic part. A developer tool that needs a weekend of setup before the first useful memory is a tool most people abandon. Getting capture working first, with defaults, and letting power users tune later is the right onboarding order for infrastructure.

The operator tax

Everything above is the upside. The honest accounting: self-hosting is a job. Backups, updates, debugging why the embeddings stopped working after an LM Studio update, making the server reachable when you are away from your desk, all yours. If the machine hosting it is off, your memory is offline, and no local system fixes that without extra infrastructure.

There is also a dependence that no architecture removes: the memory is only as good as what gets written to it. An assistant that never calls the store tool produces an empty database, however elegant the schema. MCP gives the model the tools; the model's judgment, plus your prompting, decides whether they get used.

When to build this, when to buy it

Build (or adopt) the self-hosted stack when the constraints fit: you run local models already, your workflow is developer-shaped (code, projects, debugging history), you want file-level ownership of your data, and you are comfortable being the operator. The architecture here is genuinely instructive even if you never run it: split stores by purpose, distill raw logs into curated memories, retrieve by meaning, keep the data in files you can read.

If the operator tax is the blocker, if you want memory that follows you across phone and laptop with no maintenance, the managed route exists. I work on Vilix AI, a managed MCP memory layer built on the same idea (one shared store, semantic retrieval, per-user isolation, inspect and export anytime), minus the part where you maintain the server. Either way, the lesson from the self-hosted stack stands: your context should be infrastructure you control, not exhaust that evaporates between sessions.