A memory agent that remembers you after the process dies — short-term buffer + vector recall + compression, no vector DB

#ai #agents #llm #python

Project 5 of my "Agentic AI from Zero" series is a memory-enabled conversational agent — and the whole point is this: you tell it a fact in one process, kill the process, start a brand-new one, and it still knows.

No LangChain, no Pinecone, no paid embedding API. Just Python, a free 8B model (Llama 3.1 via NVIDIA NIM), and about 200 lines. Here is how the four pieces fit together.

1. Short-term buffer (verbatim, bounded)

The last few turns are kept word-for-word so the model has immediate conversational context. It's a simple deque with a token budget — recent turns are cheap and high-signal, so they stay raw.

2. Long-term vector recall (hand-rolled, no DB)

Older facts get embedded and stored for semantic recall. The embedder is deliberately dependency-free — a stable hashing embedder:

class HashingEmbedder:
    def embed(self, text):
        vec = [0.0] * self.dim
        for tok in text.lower().split():
            h = int(hashlib.md5(tok.encode()).hexdigest(), 16)
            vec[h % self.dim] += 1.0 if (h >> 8) & 1 else -1.0
        return l2_normalize(vec)   # cosine == dot product after this

Signed buckets keep it from collapsing to all-positive, L2-normalization turns cosine similarity into a plain dot product. Retrieval is argmax over stored vectors. Load-bearing recall, zero infrastructure.

3. Context compression (on overflow)

When the short-term buffer overflows, the agent doesn't just drop old turns — it asks the model to summarize them into one compact memory note, then stores that in long-term. In the real recorded run this fired twice in session 1: the raw turns "I'm allergic to peanuts" collapsed into memory #6 — "Devanshu is allergic to peanuts." — freeing ~80 tokens while keeping the fact.

4. Relevance scoring (inject only what matters)

Every user turn triggers a retrieval. Only memories above a similarity threshold get injected into the prompt — the rest are skipped, so the context stays lean:

[INJECT] #6 (0.408) Devanshu is allergic to peanuts.
[INJECT] #2 (0.354) ...
[skip]   #4 (0.121) ...

The cross-session proof

The memory store persists to a plain memory_store.json. The recorded run is two genuinely separate OS processes:

Session 1 (process A): tells the agent its name, an allergy, and a project detail; the buffer overflows; compression fires; process exits.
Session 2 (a fresh python run.py session2): asks "what am I allergic to?" → answers peanuts by recalling memory #6 at score 0.408. Asks about the project → recalls the Safar / Flutter + Supabase summary at 0.500.

The model is only ever used for two things: writing the compression summaries and the final answer. Storage, embedding, retrieval, and relevance scoring are all deterministic Python — which is exactly what makes the memory auditable instead of magic.

Real recorded transcript (with the compression events and the per-memory relevance scores) + full code: