DEV Community: HemantAcharya

How I gave a self-hosted AI agent long-term memory, without a vector database

HemantAcharya — Mon, 03 Aug 2026 19:33:09 +0000

TL;DR: I added long-term memory to a self-hosted AI agent. The memory lives on the user's own server — no cloud, no vector database. Embeddings are stored as float32 blobs in SQLite and compared with a plain numpy cosine similarity. The easy part was storing memories; the hard part was consolidating them (add / update / delete), and two LLM bugs taught me the most.

The constraint that shaped everything

Most "AI memory" tutorials start by spinning up a vector database and shipping your data to someone's cloud. I couldn't do either.

The AI agent I work on AIDA by Autafy is self-hosted — it runs on the user's own server, under their own API key. Its whole point is that your data never leaves your hands. So when I added memory, the memory had to live there too: on your box, in a file you can open, back up, or delete. That one constraint decided the whole design.

Two kinds of memory: semantic and episodic

Most memory features only store facts — "prefers concise answers," "works in real estate." Useful, but flat. I wanted two layers:

Semantic memory — durable facts: preferences, projects, the people you work with.
Episodic memory — the moments worth remembering: "was excited about landing their first customer."

The episodic layer is what makes an assistant feel like it knows you, instead of just holding a profile card. Both are just rows in SQLite.

Why no vector database?

At per-user scale, a vector database is overkill. One person accumulates a few hundred to a few thousand memories — not millions. So embeddings get stored as raw float32 blobs right next to the memory text, and retrieval is a brute-force cosine similarity in numpy:

python
import numpy as np

memories store their embedding as a float32 blob in SQLite

def cosine_search(query_vec, rows, top_k=3):
q = query_vec / np.linalg.norm(query_vec)
scored = []
for mem_id, blob in rows:
v = np.frombuffer(blob, dtype=np.float32)
scored.append((mem_id, float(np.dot(q, v / np.linalg.norm(v)))))
scored.sort(key=lambda x: x[1], reverse=True)
return scored[:top_k]

At a few thousand rows this runs in well under a millisecond. What you get for free by skipping the vector database:

No second store to sync — one SQLite file is the memory.
Isolation — memory embeddings never touch the document/RAG store; different file, different code path.
Ownership is literal — the entire memory is one portable file the user controls, not rows in a database they don't.

The hard part: consolidation, not storage
Anyone can append rows. The real problem is the tenth conversation, where you contradict yourself.

"My favorite color is green." → store it.
Later: "actually I hate green, blue's my color." → this should rewrite the row, not add a second one.
"I'm not vegetarian anymore." → this should delete the fact, not store a weird negation.

So a nightly job (a cron sweep — never on the hot path of a chat) reviews each day's conversations, extracts candidate facts, and asks an LLM to decide against the existing memories:

Given a NEW fact and up to 3 EXISTING memories, choose ONE:

ADD → genuinely new information
UPDATE → same subject, changed info; rewrite that memory
DELETE → new fact contradicts and replaces nothing
NOOP → duplicate or not worth keeping

Every decision — the op and the distances to the nearest existing memories — gets logged, so thresholds can be tuned later from real data instead of guessed up front.

Two bugs that taught me the most

1. The model that silently returned nothing. Consolidation kept defaulting to ADD — contradictions never deleted, updates never updated. The cause: I was running a cheap reasoning model with max_tokens capped low. Reasoning models spend tokens thinking before they answer, so the budget got burned on reasoning, the response came back truncated to nothing, my JSON parse failed, and the fallback quietly chose ADD. Three sweeps ran that way because nothing errored. Lessons: give reasoning models room, log the full API response body when debugging, and never let a fallback hide a failure.

2. The feedback loop. The extractor started reading the assistant's own replies as new facts — memory-informed answers getting laundered back into memory as fresh "facts," and even user questions becoming false facts. The fix was one load-bearing rule in the extraction prompt: only the user's statements count as a source. Assistant turns are never facts.

Memory you can audit and delete

Because this runs on the user's own server, the pitch isn't "trust us with your data" — it's auditability. So every control ships with it: memory is off by default, every memory is visible and editable, each one shows its source, there's an incognito mode per conversation, and a forget-everything button. Memory without control is just surveillance.

Takeaways
For per-user memory at human scale, SQLite blobs + numpy cosine beat a vector database on simplicity and portability.
Storing memories is easy; consolidating them (add / update / delete / noop) is the real work.
Beware silent fallbacks — a parse failure that defaults to the harmless-looking action can hide a broken pipeline for days.
If your extractor reads a conversation, make sure it can't treat the assistant's own words as ground truth.

This is built into AIDA, a self-hosted AI agent at autafy.ca — but the approach works for any assistant where you'd rather own the memory than rent it. I'd genuinely like to compare notes on consolidation; it's the part I'm least sure I've nailed.

I built a self-hosted AI assistant that runs on your own server — here's why

HemantAcharya — Wed, 08 Jul 2026 23:13:38 +0000

I came out of the oil patch — years of non-destructive testing, not software. A year back I started a small automation agency called Autafy AI Automation, mostly building n8n workflows for people, the kind of behind-the-scenes automation that quietly saves hours every week.

But I kept running into the same wall. The tools were scattered. A workflow here, an integration there, an AI model somewhere else. Even when the automation worked, the person I built it for often didn't really understand what it did, or what else was possible. They had power they couldn't see or reach.

So I wanted to build something different: one place where all the tools live together, that a normal person can actually use, where you can see what the AI agent is doing and start imagining what more you could do with it.

That became AIDA - "Autafy's Intelligence Driven Agent"

What it is

AIDA is a personal AI agent you run on your own server. You install it with one command, and manage everything from a web dashboard — no terminal, no code after setup.
Here's a 2-minute demo of it actually doing things:

Once it's running, it does the kind of work that eats your day:

It sends me a morning briefing on Telegram before I even wake up — my calendar, the emails that actually need a reply, the weather, my tasks for the day. I roll over, check my phone, and I already know what my day looks like.
It triages my inbox all day, flagging what's urgent based on rules I set, and can even draft the reply.

It answers questions from my own documents — upload a PDF or a spreadsheet, ask a question, and it answers with a citation back to the source.
It connects to the tools I already use — Gmail, Calendar, Drive, Outlook, Notion, Slack, Stripe, and more.
And it runs on whatever AI model I want — including fully local models, so nothing has to leave my machine at all.

Who it's for

Honestly, most working professionals.

Solo entrepreneurs and freelancers who are doing the work and running the business, and don't have time to babysit their inbox. Consultants who live out of their calendar. Even students — you can drop your study materials into the knowledge base and ask the agent questions, and it answers from your own notes.

It's not built for one narrow niche. It's built for anyone who has more to do than hours in the day and wants a capable assistant that's actually theirs.

Why self-hosted, and why pay once

This is the part I care about most.

Most AI assistants run on someone else's servers. That means your email, your files, your calendar — your clients' data — all pass through a company you have to trust. I didn't want that, and I didn't think my clients should have to accept it either.

So AIDA runs on your server. Your data stays with you. The only thing that ever leaves is the call to whichever AI provider you choose and if you run a local model, not even that.

And it's a one-time price, not a subscription. You buy it once and own it for life. I'd rather have customers who feel like they own something than customers I'm billing forever.

The honest part

It took a few months to build, mostly solo. The hardest part wasn't the AI, it was the plumbing. Wiring up all those tools and integrations so they work together reliably, from one place, took far longer than I expected. Every service has its own quirks, its own auth, its own edge cases. Making it feel simple on the outside meant a lot of unglamorous work on the inside.

But that was the whole point. The value isn't any single feature, It's that everything finally lives in one place a normal person can actually use.

Try it

AIDA is live now. It installs on any small VPS with one command, there's a free trial (no credit card), and it's a one-time founding license for the first customers before the price goes up.

If you've ever wanted a capable AI assistant that runs on your own machine and keeps your data yours, I'd genuinely love for you to try it and I'm around to answer any questions.

https://autafy.ca