Every AI agent I used had the same two failures. It forgot everything the moment I closed the chat, and it confidently invented facts it didn't have. I didn't want a cloud service holding my documents, and I didn't want to fork the agent. So I built two plugins.
This is the story of the design decisions behind them: why hybrid search, why "quote or refuse" belongs in code and not in a prompt, why I threw out PyTorch on purpose, how a version-keeping memory (SUPERSEDE) beats overwriting, and how you can run the whole thing without a cloud key at all, local ONNX embeddings included. Same direction the big vector-search and agent-memory projects took, packaged as one local file.
The two failures
Almost every "agent memory" I tried let me down in one of two ways.
First one: it doesn't actually remember. The agent I run ships with a memory that's two tiny files, a few hundred tokens each. Fill them up and it hands you an error instead of compacting. And nothing gets written unless the model decides, mid-conversation, to call a tool. So my decisions, my corrections, the way I like things done? Gone by default. New session, and yesterday never happened.
Second one: it doesn't know my documents, but it'll answer about them anyway. There was no real knowledge base. Files I dropped in went into a cache and got wiped after 24 hours. Ask a plain LLM about a document and you've got two bad options. Paste the whole thing into the prompt (expensive, and it gets lost in the middle), or lean on its general knowledge and watch it make things up with a straight face.
I didn't want to patch the core and babysit a fork forever. So I wrote two plugins. They hang off the agent's plugin API and touch zero lines of its source.
MemoHood remembers the conversation. It notices what matters, pulls it back before the next reply, and decides on its own when an old fact needs replacing.
MemoBase is a private knowledge base over your files, web pages, YouTube, audio, and Obsidian notes. It answers only from your sources, with an exact quote, or it tells you straight that the answer isn't there.
One SQLite file each. Both run fully local if you want. The rest of this is the why behind every call I made.
What it looks like
Two moments that show what these are for.
Memory across sessions. Monday I mention, offhand: "By the way, I'm allergic to penicillin." A month later, brand-new chat: "Suggest an antibiotic for a sore throat." It answers around the allergy. I reminded it of nothing. That's MemoHood. The fact got captured for free (an allergy is an obvious keeper), pinned so it never decays, and pulled back automatically before the reply.
Answers you can trust, or a clean no. I drop in a contract PDF and ask about its term. Back comes the answer with the exact clause quoted and the page number. Then I ask about something the contract doesn't cover. No improvising. It says the answer isn't in the base. That's MemoBase, and the honesty isn't a personality I prompted for. It's enforced in code. Which is the whole point of the next two sections.
Why hybrid search (FTS5 + vector + RRF)
Every retrieval system has to pick how it finds things. The two obvious options each fail in a different direction.
Keyword search (full-text, BM25) is literal. Ask about "the contract" and it won't find a chunk that only says "this agreement." Same meaning, different words, no match. But it's unbeatable on exact tokens: error codes, SKUs, function names, someone's surname.
Vector search is the opposite. It matches by meaning, so "agreement" and "contract" sit close together. And it loses exact tokens. Search for an order number like A-4471 and semantic similarity just shrugs, because a string like that has no "meaning" to be near.
Pick one and you eat the other's blind spot. So I run both and fuse the results with Reciprocal Rank Fusion, the same trick the big search stacks use. RRF is beautifully dumb: it ignores the raw scores (which aren't comparable across two systems anyway) and combines by rank.
Two ranked lists: one from FTS5/BM25, one from the vector index.
RRF fuses by position, not by raw score. k=60 is the standard constant.
def rrf(rank, k=60):
return 1 / (k + rank)
A chunk that's #2 in keyword search AND #5 in vector search
outranks one that's #1 in a single list but absent from the other.
score = rrf(fts_rank) + rrf(vec_rank)
A chunk that shows up strongly in both lists wins. That's exactly what you want: found by meaning and by the exact term. If a Cohere key is present, the top candidates get one rerank pass on top. No key? It falls back to RRF order and keeps going instead of erroring.
Best part: it all lives in one SQLite file. Full-text is FTS5 (with Russian stemming on its own column, so "договора" finds "договор"). The vectors sit in sqlite-vec, same database. No separate vector server, no external index to babysit. One file you can copy, back up, or delete.
Why "quote or refuse" belongs in code, not in a prompt
You can ask a model to answer only from the sources and to admit when it doesn't know. People write long system prompts doing exactly that. It works, right up until the one time it doesn't. And you've got no way to tell the honest answers from the confident fakes.
A prompt is a request. I wanted a guarantee. So MemoBase runs a four-stage loop where the anti-hallucination part is baked into the structure, not asked for politely:
- sufficiency gate -> did search return a confidently relevant chunk? if not, refuse now, before the model generates anything
- tool-less answer -> the model answers with ONLY the retrieved chunks in hand, no other tools attached, nothing to "fill in" from
- citation check -> every quote in the answer is matched against the source chunk verbatim (exact, else fuzzy); a fabricated or inexact quote is dropped
- honest refusal -> if no verified citation survives, return "not in the base" instead of unverified text The citation check is the load-bearing piece. The model can claim whatever it likes. Every quote still gets diffed against the raw source text. Not word-for-word? Stripped. If nothing survives, you get the refusal, not a smooth sentence with nothing under it. Lying isn't discouraged here. It just isn't available.
That's also why a tool-less model writes the answer. Give it web access during generation and "answer only from these chunks" becomes a suggestion it can route around. Take the tools away and the fragments are all it physically has.
Why I threw out PyTorch on purpose
The default RAG stack is heavy. A typical embedding setup drags in PyTorch, then gigabytes of model weights, and it often ships under AGPL, a license that can force you to open-source your own code if you build on it. None of that fits on a $5 VPS. And I didn't want a license lawyer living in my dependency tree.
So the rule was simple: standard library plus a few light packages, installs in seconds. The vector index is sqlite-vec. Stemming is PyStemmer. The offline embedder, when you want one, is fastembed, which runs the model through ONNX Runtime. No PyTorch anywhere, no AGPL, nothing to compile.
The payoff is boring and I love it. It drops onto a cheap box, starts fast, and there's nothing exotic to break on someone else's machine.
Local or cloud, and no keys required
At install you pick a fork in the road. Both ends are real, not one "proper" path and one toy.
Cloud gives you the most quality for the least setup: embeddings via Cloudflare Workers AI (BGE-M3), an optional Cohere rerank, and the extra sources (YouTube captions, audio transcription). Most of those have a free tier. There's also a per-provider monthly spend ceiling, so a runaway job can't hand you a surprise bill.
Fully local is why I bothered building the ONNX path. Reinstall with one flag and you get fastembed plus a multilingual model (~2.2 GB, downloaded once). After that: no keys, nothing to pay.
./install.sh --local # Linux / macOS
.\install.ps1 -Local # Windows
config.yaml — one embedder, shared by both plugins
memory:
provider: memohood
memohood:
embedder: { provider: local, model: intfloat/multilingual-e5-large, dims: 1024 }
memobase:
embedder: { provider: local, model: intfloat/multilingual-e5-large, dims: 1024 }
On privacy: even in cloud mode, the only thing that leaves your machine is the text of your question and the candidate chunks, for embedding and reranking. The source files and the rest of the database never get uploaded. In local mode nothing leaves at all.
A memory that keeps versions: SUPERSEDE and the gate
MemoHood is where I spent the most thought, because "remember facts" is the easy half. The hard half is what happens when facts change.
Most memories overwrite. You said the deadline is Friday, then you say Monday, and Friday is gone. Fine, until keyword search drags the stale summary back up and the agent "corrects" you with last week's plan. I've watched it happen. So MemoHood doesn't overwrite. It supersedes.
New fact contradicts an old one? The new one goes on top. The old one gets marked stale, stamped with a date, and kept in the record's history. Normal search only shows the current version, but the old one is one call away whenever you want it. The memory always knows both "how it is now" and "how it was," and it never quietly rewrites your past.
Two more decisions keep it cheap and quiet.
The capture path is free first. Explicit signals (a correction, a decision, a flat "remember this") get caught by a keyword scorer with zero LLM calls. Only genuinely borderline turns cost one call to a cheap model, which decides whether they're worth keeping. You don't pay to remember the obvious stuff.
And there's a gate before recall. A tiny offline static-embedding model (Model2Vec, no network, no key) judges whether an incoming turn is even a memory question. A plain "thanks" doesn't trigger a search. The rule leans cautious on purpose: it skips recall only when the small-talk signal clearly wins, because a missed recall is worse than an extra one.
Durable stuff (a name, a birthday, an allergy) gets pinned and skips the nightly forgetting curve entirely. Everything else decays gently over time, fast for fleeting events, slow for stable facts, in a background job that never sits on the reply's hot path.
I didn't invent any of this, and that's good
Fair question: did I actually invent anything here? Mostly no. And that's the reassuring part, because it means the design points the same way as teams with a lot more money. I just packaged it local and in one file.
Hybrid search fused with RRF is what Microsoft ships in Azure AI Search (the retrieval under Copilot), Google in Vertex AI Search, Amazon in OpenSearch, Elastic in Elasticsearch. Big cloud engines. I put the same idea in a file on disk.
Answer only from sources, with a citation? That's Perplexity and Google's NotebookLM. Cloud products. I did the local, free version.
Fact-extracting memory is a whole field now. mem0 pulls facts through a model the same way, and it works well enough that AWS made it the default memory in Bedrock AgentCore. Zep keeps memory as a temporal graph and doesn't erase the old state, which is my SUPERSEDE in spirit. Letta (ex-MemGPT) treats memory like an operating system. Even the giants shipped it: OpenAI has ChatGPT Memory, Anthropic rolled out Claude Memory that builds facts from history. The idea is everywhere. The particular assembly (both halves, local, one SQLite file, zero core changes) is mine.
Limitations, honestly
No tool is all upside. Here's what I'd want to know before installing.
Old history isn't back-filled. MemoHood extracts facts from turns after you install it. It can index existing history for recall, but it won't mine your old chats for new facts.
Cost estimates are approximations. The spend numbers come from providers' public pricing, not a guaranteed invoice. The monthly ceiling is real, but treat the figures as best-effort.
Transcription timecodes vary by path. The primary Whisper path gives real decoder timecodes. The fallback path estimates them and drifts on long audio, and those transcripts get flagged lower-trust.
No one-click social ripper. A page behind a link gets ingested as text. Pulling a specific Instagram or TikTok video means downloading the file yourself and feeding it as video.
Restore is manual. Nightly backups are ready-to-use database files, but there's no one-command restore yet.
How to install
Two paths. You choose by what you want.
Path A, fully local and free. Not a single paid key. The only cost is downloading the embedding model once.
./install.sh --local # Linux / macOS
.\install.ps1 -Local # Windows
The installer finds the agent's virtual environment, installs the light dependencies, copies each plugin into place, and wires up the config. Restart the agent and both plugins are live.
Path B, cloud, for maximum quality. A setup wizard walks you through the keys one at a time (Cloudflare for embeddings, Cohere for rerank, plus optional keys for YouTube and audio), checks each with a live request, and stores them masked. Skip any service you don't need. The matching feature falls back quietly instead of crashing.
That's the whole thing. Two plugins, one SQLite file each, no fork, and a local mode with nothing to pay and nothing leaving your machine.
Both repos are MIT:
MemoHood (in-chat memory): https://github.com/mxskorohood-cmd/memohood
MemoBase (knowledge base): https://github.com/mxskorohood-cmd/memobase
Top comments (0)