Your Agent's Memory Is a Lie: A Durable, Queryable Memory Layer for Local LLM Agents

You've seen the tutorial. Somewhere in the agent class there's a line like:

self.memory = []

It looks harmless. It works in the demo. Then it quietly betrays you the moment the process restarts, the context window fills up, or you try to remember something you stored five minutes ago.

This post builds a real memory layer for local LLM agents: one that survives restarts, does semantic recall instead of exact-string matching, and needs zero cloud and zero separate vector database. The whole thing runs on SQLite + a tiny in-process SQLite extension + your already-running Ollama.

The lie: self.memory = [] is not memory

An in-memory Python list fails on four axes that actually matter in production:

What you need	self.memory = []	Reality
Persistence	❌ Gone on restart	A reboot wipes everything
Semantic recall	❌ Exact match only	You can't "find similar," only if x in list
Scalable retrieval	❌ Linear scan	Dumping the whole list back is a context bomb
Metadata / scoping	❌ None	No "when," "what kind," or "whose"

The fix isn't complicated. Memory is just two operations:

1. Store text as an embedding (a vector) + the raw text + metadata, on disk.
2. Retrieve by finding the nearest vectors to a query embedding (nearest-neighbor search).

If you can do those two things locally, you have a memory layer. Here's how, with no servers.

The stack (all local, all free)

• SQLite — durable storage. One file on disk. Survives restarts by definition.
• sqlite-vec — an in-process SQLite extension that adds a vec0 virtual table for vector similarity search. No Postgres, no Qdrant, no Pinecone. docs
• Ollama — generates embeddings locally with a model like nomic-embed-text (768-dim). docs

Why not a "real" vector DB? For a solo agent or even a small team agent, you're storing thousands, not billions, of vectors. sqlite-vec does brute-force + index KNN fast enough at that scale and adds zero operational overhead. Graduate to a dedicated vector DB only when you actually hit millions of vectors.

Step 0: Install and pull the model

pip install sqlite-vec ollama
ollama pull nomic-embed-text   # 768-dimensional embeddings, runs locally

nomic-embed-text produces 768-dim vectors. The dimension in your vec0 table MUST match the model, or inserts fail. (Other local options: mxbai-embed-large = 1024-dim, embeddinggemma = 768-dim.)

Step 1: The schema

We keep the raw text + metadata in a normal memories table, and the vectors in a vec_memories vec0 table keyed by the same id. Keeping them separate is the recommended pattern — it lets the vector index stay compact and the text stay queryable.

import sqlite3, sqlite_vec
from sqlite_vec import serialize_float32

db = sqlite3.connect("agent_memory.db")  # a file on disk -> survives restarts
db.enable_load_extension(True)
sqlite_vec.load(db)          # load the vector extension into THIS connection
db.enable_load_extension(False)

db.execute("""
CREATE TABLE IF NOT EXISTS memories (
    id         INTEGER PRIMARY KEY,
    content    TEXT NOT NULL,
    mem_type   TEXT,            -- 'fact' | 'preference' | 'event' | 'skill'
    created_at TEXT DEFAULT (datetime('now')),
    importance REAL DEFAULT 0.5
)
""")

# cosine distance: Ollama returns L2-normalized vectors, so cosine == L2 ordering,
# but we set it explicitly so the intent is unambiguous.
db.execute("""
CREATE VIRTUAL TABLE IF NOT EXISTS vec_memories USING vec0(
    id        INTEGER PRIMARY KEY,
    embedding float[768] distance_metric=cosine
)
""")

Step 2: Embedding helper (Ollama, local)

Ollama's modern embeddings endpoint is POST /api/embed (the old /api/embeddings is deprecated). The Python SDK wraps it:

import ollama

def embed(text: str) -> list[float]:
    # returns a single vector for a single string
    return ollama.embed(model="nomic-embed-text", input=text)["embeddings"][0]

Verify it works:

v = embed("The build failed because port 5000 was already in use.")
print(len(v))   # 768

The response is L2-normalized (unit length) by Ollama, which is exactly what cosine similarity wants.

Step 3: Store a memory

def remember(content: str, mem_type: str = "fact", importance: float = 0.5):
    vec = embed(content)
    with db:
        cur = db.execute(
            "INSERT INTO memories(content, mem_type, importance) VALUES (?, ?, ?)",
            (content, mem_type, importance),
        )
        row_id = cur.lastrowid
        db.execute(
            "INSERT INTO vec_memories(id, embedding) VALUES (?, ?)",
            (row_id, serialize_float32(vec)),   # MUST serialize for vec0
        )
    return row_id

Two things to internalize:

• Always serialize_float32 the vector before handing it to vec0. JSON strings work in raw SQL but the Python vec0 MATCH path expects the compact float32 bytes. The sqlite_vec helper exists for this exact reason.
• Store the raw text separately. The vector is for finding; the text is what you actually feed the model. Never throw away the original.

Step 4: Query by meaning (not by exact string)

This is the whole point — semantic recall:

def recall(query: str, k: int = 5):
    qvec = serialize_float32(embed(query))
    rows = db.execute(
        """
        SELECT m.id, m.content, m.mem_type, m.importance, v.distance
        FROM vec_memories v
        JOIN memories m ON m.id = v.id
        WHERE v.embedding MATCH ?
          AND k = ?
        ORDER BY v.distance
        """,
        (qvec, k),
    ).fetchall()
    return rows

The WHERE embedding MATCH ? AND k = ? ORDER BY distance is the KNN query form sqlite-vec recognizes. You pass the serialized query vector as the MATCH argument.

Watch it work:

remember("Deploy broke because the DB migration ran before Postgres was ready.")
remember("User prefers concise Hinglish explanations, not formal English.")
remember("The /api/orders endpoint times out above 2k req/s.")

hits = recall("why did production go down last night?")
for h in hits:
    print(round(h[4], 3), "|", h[2], "|", h[1])
# 0.312 | fact | Deploy broke because the DB migration ran before Postgres was ready.
# 0.481 | fact | The /api/orders endpoint times out above 2k req/s.
# 0.902 | preference | User prefers concise Hinglish explanations, not formal English.

Notice: the query never contained the words "deploy," "migration," or "Postgres" — yet the right memory surfaced first. That's semantic recall, and it's why this beats if x in self.memory.

Step 5: Filtering with metadata (the part lists can't do)

vec0 supports metadata columns you can constrain inside the KNN query. Let's add one so we can ask "only preferences" or "only this user's memories":

db.execute("DROP TABLE IF EXISTS vec_memories")
db.execute("""
CREATE VIRTUAL TABLE vec_memories USING vec0(
    id        INTEGER PRIMARY KEY,
    agent_id  TEXT,                                -- metadata column, filterable
    embedding float[768] distance_metric=cosine
)
""")

Now scope by agent:

def recall_scoped(query: str, agent_id: str, k: int = 5):
    qvec = serialize_float32(embed(query))
    return db.execute(
        """
        SELECT m.content, v.distance
        FROM vec_memories v
        JOIN memories m ON m.id = v.id
        WHERE v.embedding MATCH ? AND k = ?
          AND v.agent_id = ?
        ORDER BY v.distance
        """,
        (qvec, k, agent_id),
    ).fetchall()

This is how you stop one agent's memories from leaking into another's context — something an in-memory list makes painfully easy to get wrong.

Caveats & trade-offs (read before you ship)

• sqlite-vec is pre-v1. The author explicitly warns of breaking changes. Pin it: pip install sqlite-vec==0.1.9 (latest stable as of this writing; 0.1.10 is still alpha) and re-test on upgrades.
• Dimension mismatch is a hard failure. float[768] must equal your model's dims. nomic-embed-text→768, mxbai-embed-large→1024. If you swap models, recreate the vec0 table.
• Embedding quality is the ceiling. Retrieval is only as good as the vectors. A weak local model will surface weak results. nomic-embed-text / mxbai-embed-large are solid defaults; don't expect OpenAI-grade recall from a tiny model.
• Cold-start latency. The first embed call loads the model into VRAM/RAM (~seconds). Set keep_alive on the Ollama request if your agent queries in bursts.
• Scale limit. Brute-force/index KNN in sqlite-vec is great to low-millions of vectors. Past that, move to a dedicated vector DB.
• Chunk long texts. Embeddings compress a whole string into one vector. A 5-page doc as one embedding loses detail. Chunk first, store each chunk, recall the best chunks.
• No built-in forgetting. Add your own: a periodic DELETE FROM memories WHERE importance < ? AND created_at < ? (and the matching vec_memories row), or summarize old memories into a single "digest" memory.

Wiring it into an agent

Minimal loop:

def answer(agent, user_msg):
    context = "\n".join(r[0] for r in recall(user_msg, k=5))
    reply = agent.generate(f"Memory:\n{context}\n\nUser: {user_msg}")
    remember(f"User: {user_msg}\nAgent: {reply}", mem_type="event")
    return reply

You retrieve relevant past context, generate, then persist the exchange. Memory now compounds instead of resetting every reboot.

Takeaways

• self.memory = [] is a demo hack, not infrastructure. It loses everything on restart and can't recall by meaning.
• Real memory is just store an embedding + retrieve nearest neighbors — and you can do both locally.
• SQLite + sqlite-vec + Ollama gives you durable, semantic, scoped, metadata-rich memory with zero servers and zero cloud bills.
• Pin sqlite-vec, match your vector dimension to your model, chunk long texts, and add a forgetting policy.
• Scale up to a real vector DB only when you actually outgrow millions of vectors. Most agents never will.

The "context is the new bottleneck" isn't hype — it's the part of your agent that actually decides whether it's brilliant or useless. Stop pretending a Python list is memory, and give it a real one.

References

• Ollama Embeddings API (/api/embed) — https://docs.ollama.com/api/embed
• Ollama embeddings usage (CLI + Python ollama.embed) — https://github.com/ollama/ollama/blob/main/docs/capabilities/embeddings.mdx
• sqlite-vec (vector search SQLite extension) — https://github.com/asg017/sqlite-vec
• sqlite-vec vec0 virtual tables — https://alexgarcia.xyz/sqlite-vec/features/vec0.html
• sqlite-vec KNN queries — https://github.com/asg017/sqlite-vec/blob/main/site/features/knn.md
• Signal: "Context is the new bottleneck" in 2026 agent tooling — https://dev.to/felixwang007/the-ai-race-just-left-the-model-layer-30-days-of-github-trending-data-proves-it-1lm7