Inside the Memory Decision Loop: How AI Agents Decide What to Remember, Update, or Forget

Most people who add "memory" to an AI agent do the same thing: embed every message, throw the vector into Pinecone or pgvector, and call similarity_search at query time. It works for a demo. It falls apart in production, because nothing ever updates or deletes anything — the store only grows, and it fills with contradictions.

Say a user tells your agent "I live in Austin" in March and "I just moved to Denver" in July. A naive vector store keeps both. At retrieval time, both come back as top-k matches for "where do I live," and now your LLM is holding two contradictory facts with no signal about which one is current. This is the actual problem memory layers like Mem0 are built to solve, and the mechanism is more interesting than "vector DB with extra steps." Here's how it actually works.

The pipeline has four stages, not one

A naive RAG setup has one stage: embed and store. A real memory layer has four:

1. Extraction — turn a raw conversation turn into candidate facts
2. Retrieval — find existing memories that might relate to each candidate
3. Decision — for each candidate, decide ADD, UPDATE, DELETE, or NOOP against what's already stored
4. Consolidation — write the resolved state back, not just append to it

The first stage is the one everyone builds. The second and third are the ones that make memory actually useful instead of a growing pile of unreconciled embeddings.

Stage 1: extraction is a compression step, not a copy step

You don't embed the raw message "yeah so I just moved to Denver last week, still unpacking boxes everywhere." You run it through an LLM extraction prompt that pulls out the durable fact: user.location = Denver. Everything about boxes and unpacking is conversational noise — useful for the current turn, useless six weeks from now.

EXTRACTION_PROMPT = """
Extract durable facts about the user from this message.
Ignore small talk, emotional tone, and one-off requests.
Return a list of atomic facts as short subject-predicate-object statements.

Message: {message}
"""

candidates = llm.extract(EXTRACTION_PROMPT.format(message=turn))
# -> ["user lives in Denver"]

Atomic matters here. "User lives in Denver and works remotely and has a dog" should become three separate candidate facts, not one blob — because each one might independently need to be added, updated, or deleted later, and you can't do that to a fact if it's welded to two unrelated ones.

Stage 2: retrieval finds what might conflict, not what's relevant

This is the part that's easy to get backwards. At write time, you're not doing similarity search to answer a question — you're doing it to find memories that the new candidate might contradict or refine. So for "user lives in Denver", you embed the candidate and pull the top few nearest existing memories:

candidate_embedding = embed("user lives in Denver")
neighbors = vector_store.search(candidate_embedding, k=5)
# -> [("user lives in Austin", score=0.89), ("user has a dog", score=0.31), ...]

The high-scoring neighbor (user lives in Austin, 0.89 similarity) is exactly the case that matters — semantically close enough to be about the same fact, but not identical. That's the conflict the next stage has to resolve.

Stage 3: the actual decision — this is the part people skip

For each candidate + its neighbors, a second LLM call (or in leaner implementations, a classifier) decides what to do. Mem0's public writeups describe exactly this operation: given a new fact and its nearest existing memories, output one of ADD, UPDATE, DELETE, or NOOP.

DECISION_PROMPT = """
New fact: {candidate}
Existing related memories: {neighbors}

Decide the operation:
- ADD: new fact, no real overlap with existing memories
- UPDATE: new fact supersedes an existing memory (same subject, changed value)
- DELETE: new fact explicitly contradicts and invalidates an existing memory
- NOOP: new fact is already captured, do nothing

Return: {{"operation": ..., "target_id": ..., "resolved_fact": ...}}
"""

decision = llm.decide(DECISION_PROMPT.format(
    candidate="user lives in Denver",
    neighbors=[("m_204", "user lives in Austin")]
))
# -> {"operation": "UPDATE", "target_id": "m_204", "resolved_fact": "user lives in Denver"}

This is the step that turns a vector store into a memory system. Without it, you have an append-only log with a fuzzy search index bolted on. With it, you have something closer to a mutable key-value store where the "key" is semantic similarity instead of an exact string.

The failure mode to watch for: an overly aggressive DELETE/UPDATE threshold merges facts that only look similar. "User lives in Denver" and "user was born in Denver" are 0.85+ cosine similar and mean completely different things. This is why the decision step needs the LLM to reason about semantics, not just a similarity score cutoff — a pure threshold rule (if score > 0.8: overwrite) will silently corrupt memory in exactly the cases where getting it right matters most.

Stage 4: consolidation — write once, not append

The resolved operation gets applied atomically: an UPDATE overwrites the existing vector and metadata for m_204 (new embedding, incremented version, updated timestamp) rather than inserting a new row and leaving the old one to rot. A DELETE tombstones the memory instead of a silent removal, so you can audit why something disappeared if a user asks "wait, didn't I tell you I live in Austin?"

if decision["operation"] == "UPDATE":
    vector_store.update(
        id=decision["target_id"],
        embedding=embed(decision["resolved_fact"]),
        text=decision["resolved_fact"],
        version=existing.version + 1,
        updated_at=now(),
    )

Retrieval-time scoring isn't just similarity either

Once memories are clean, the query-time ranking still shouldn't be pure cosine similarity. Production memory layers blend it with recency and access frequency, because a fact from an hour ago and a fact from eight months ago can both be top-3 nearest neighbors, but they're not equally trustworthy:

score = (0.6 * similarity) + (0.25 * recency_decay(age_days)) + (0.15 * access_frequency)

The decay function matters more than its weight suggests — a linear decay treats a 30-day-old fact and a 300-day-old fact as almost the same, while an exponential decay (closer to how the Ebbinghaus forgetting curve actually behaves) lets stale facts fade without a hard expiry that deletes something still true.

The takeaway

If you're building agent memory and it's just store.add(embedding) on every turn, you don't have memory — you have an unindexed diary the model reads through a straw. The four-stage loop — extract, retrieve neighbors, decide the operation, consolidate — is what separates a system that gets more useful as it accumulates history from one that gets slower and more contradictory. That decision step is also the one piece you can't shortcut with a bigger embedding model or a faster vector index; it needs actual reasoning about whether two facts are the same fact, a newer version of it, or something else entirely.