Agent Memory Is Not a Vector Database. It's a Forgetting System.

Ask someone to sketch memory for an LLM app and you'll usually get one arrow.

user message -> embed -> vector DB -> search later

That's the demo version. It works for a weekend project. It also hides most of the problem.

This post is about the part the arrow leaves out: deciding what gets stored in the first place. The short version is that a good memory system spends most of its effort on forgetting.

Before we start: I haven't built this yet. This is a design walkthrough from studying how a ChatGPT-style memory system could be put together, not a report from production. I say "ChatGPT-style" on purpose. I can't see OpenAI's internals, so I'm not describing them. This is reasoning about how you'd design something that behaves like it.

The one problem memory solves

Imagine an assistant with no memory. In one chat you say you like short answers. In the next you ask it to help plan your startup, and it has forgotten all of it. You explain who you are, what you want and how you like to be spoken to. Again.

Memory exists to solve exactly one problem: information loss across interactions.

That sentence has a consequence people skip. If the goal is to reduce loss, the goal is not to store everything. A database stores everything. A memory system stores only what improves future behavior. Memory is selective retention.

Hold onto the word "selective". It means the first thing in front of your store isn't a write. It's a decision about whether to write at all.

Why one arrow is the wrong picture

A production memory system breaks into six parts. Each has its own job and its own ways to fail.

Figure 1. The one-arrow picture, and the six jobs a memory system has to do. Each box shows the job and the question it answers.

A vector store covers, at best, storage and retrieval. That leaves four of the six out of the one-arrow picture, and those four are where a lot of the hard problems live.

This post covers Capture and Evaluation, with a glance at Storage. Retrieval, decay and governance come in later posts in this series.

Memory comes in kinds

Before designing a store, look at the thing you're copying. Humans don't keep everything in one bucket. They remember in kinds, and each kind has a different lifetime, different update rules and different triggers for recall. That maps well to engineering.

Kind	What it holds	Example	Storage rule
Episodic	Events, with a time attached	The user launched a startup	Event log, append-only
Semantic	Stable facts	The user works in ML	Overwrite when the fact changes, deduplicate
Procedural	How the user likes things done	Prefers first-principles explanations	Small set, high retrieval priority
Working	The live conversation	Everything in the current context window	Not persisted by default

Two consequences. First, your store shouldn't be one table with a text column. The kind decides the rules. Second, memories move. Something said once in the live conversation, if it proves useful, gets promoted to a longer-lived kind so it survives the conversation.

The write path and the read path

Memory is two pipelines: one that writes it and one that reads it back.

Figure 2. The write path on the left, the read path on the right. The memory store is the only thing they share.

On the write path, the extractor is an LLM task. It reads a conversation turn and pulls out candidate memories: preferences, goals, projects, relationships, long-term facts. Its output is structured (JSON), not a blob of text. The evaluator decides whether each candidate is worth keeping. That's the write gate, and it gets its own section below. The store persists whatever survives, by kind.

On the read path, the retriever finds candidate memories for the new message. The context composer turns the best ones into a short block the model can read. The LLM answers with that block in its context.

The read path is where the system feels smart or feels broken, because the user only ever sees what you surface, never what you kept. That's the next post. For now, one rule: nothing reaches the read path unless it survived the write path.

The write gate

This is where most memory systems go wrong. The naive version stores everything. That causes memory pollution: the store fills with noise, retrieval drowns in it, and every prompt gets cluttered with irrelevant facts.

The fix is to score each candidate by future utility, with one question: will this improve a future conversation?

Candidate	Utility	Why
"I use PostgreSQL"	High	A stable fact that shapes future technical answers. Keep it.
"I'm building a startup"	High	Anchors a lot of future conversations. Keep it, high importance.
"Today I had coffee"	Low	True, but it changes no future decision. Drop it.

The component that asks this question is the evaluator. In an MVP it can be an LLM judge with a prompt like: "Should this memory improve future conversations? Answer yes or no, and give an importance from 1 to 10." At scale it should turn into a cheap deterministic classifier, because the same memory ought to score the same way every time. Either way, it sets two fields on the record: importance and confidence.

Confidence matters more than it looks. Say a user writes "I'm thinking about moving to Bangalore." A careless extractor stores "lives in Bangalore." Now any answer that depends on where the user lives can be wrong, and sound sure about it. That's the wrong-memory failure, and a confidence score is part of the fix.

Here's what the gate does with three candidates:

Figure 3. The write gate on three candidates. One is stored, two are dropped, for two different reasons.

And here's the same gate as a sketch in code. The judge is a stub with canned verdicts so the example runs. In a real system that's an LLM call.

from dataclasses import dataclass
from datetime import datetime, timezone


@dataclass
class Verdict:
    keep: bool          # will this improve a future conversation?
    importance: int     # 1-10
    confidence: float   # 0.0-1.0, how sure we are it is true


def write_gate(candidate, judge, min_importance=5, min_confidence=0.6):
    """Return a memory record if the candidate earns a place in the store, else None."""
    verdict = judge(candidate)
    if not verdict.keep:
        return None
    if verdict.importance < min_importance or verdict.confidence < min_confidence:
        return None
    return {
        "content": candidate,
        "importance": verdict.importance,
        "confidence": verdict.confidence,
        "created_at": datetime.now(timezone.utc),
    }


# Stand-in for the LLM judge, with canned verdicts so the example runs.
CANNED = {
    "User uses PostgreSQL": Verdict(True, 8, 0.95),
    "User had coffee today": Verdict(False, 1, 0.99),
    "User is thinking about moving to Bangalore": Verdict(True, 6, 0.40),
}

for text in CANNED:
    record = write_gate(text, CANNED.get)
    print(f"{text!r:50} -> {'STORED' if record else 'dropped'}")

Output:

'User uses PostgreSQL'                             -> STORED
'User had coffee today'                            -> dropped
'User is thinking about moving to Bangalore'       -> dropped

The thresholds are placeholders. You'd tune them against real conversations. A real system might also keep the Bangalore item at low confidence instead of dropping it. What matters is that the number exists, and that something checks it before the memory can shape an answer.

The memory record

Once the gate approves something, what lands in the store is a structured record, not a blob of text. A first cut:

memory {
  id            unique identifier
  user_id       owner, isolates one user from another
  type          episodic | semantic | procedural
  content       the fact, in plain language
  importance    how much this should weigh        (set at write)
  confidence    how sure we are it is true         (set at write)
  source        where it came from: conversation id, document, user
  created_at    when we learned it
  updated_at    when it last changed
}

Two fields do quiet work. importance and confidence let the system tell "is building a startup" apart from "maybe thinking about moving cities". source is what lets the system answer "why do you think that?" later. Each field is there because it enables a decision further down the pipeline: ranking, decay, explaining. Decay adds two more fields (a weight and a reinforcement count), which is a later post.

What to keep in mind if you build this

1. Name the problem before the tools. The problem is information loss across conversations. If a candidate memory doesn't reduce it, don't store it.
2. Put the gate in front of the store. Decide what gets in before you decide where it goes. Anything that accumulates state, like logs, caches or indexes, needs an admission policy.
3. Type your memories. Events, facts and preferences have different lifetimes and update rules. One table with one text column ignores that.
4. Store confidence and source with every record. Without them you can't catch a wrong memory, and you can't explain a right one.
5. Don't stop at a vector store. Retrieval, decay and governance each need an owner, and each fails in its own way.

If you want a short checklist to start from, the notes I'm working from open with a set of questions to answer before any design. The first three: What decision will this memory improve? What deserves remembering? What deserves forgetting? Answer those before you pick a database.

Next in the series: retrieval. Storage is mostly a solved problem. Retrieval is where the system feels smart or feels broken.