How to Build Persistent Agent Memory Across Sessions

Your agent can follow a rule perfectly at the start of a conversation and still miss it later.

Maybe the rule is simple: don't share user contact details, don't book meetings outside business hours, or don't recommend discontinued products. The agent follows it for the first few turns. Then the conversation gets longer. Tool results pile up. The user changes direction a few times. By turn 15, the original constraint is still technically in the context, but the model is no longer treating it like the most important instruction.

That failure is easy to misread. It is not always a prompt-quality problem, and it is not solved just by using a model with a larger context window. The issue is that the context window was never designed to behave like durable memory.

In this article, we'll build a two-layer memory architecture for agents: one layer for active conversation state and another for facts, preferences, and constraints that need to survive across sessions. The implementation uses Python and Actian VectorAI DB, but the pattern is framework-agnostic and can be implemented in agent frameworks such as LangChain, LangGraph, LlamaIndex, AutoGen, or Mastra.

Why the Context Window Is Not a Database

Most production agent failures come from one mistake: developers treating the context window like a database. The context window is the part of the conversation that's currently influencing the model's next answer. It looks like storage because text accumulates in it, but it behaves more like volatile working memory than durable storage in three ways that matter.

First, the context is volatile. Everything in it is gone when the session ends, no automatic persistence, no recovery. Close the chat, open a new one, and the agent has no memory of what came before.

Position also matters. The model pays more attention to information at the start and end of the context than to information stuck in the middle, a behavior documented in "Lost in the Middle." For agent memory, that translates to a predictable failure: a constraint set in the first turn drifts further from the model's focus as the conversation grows. A 2026 study at the University of Florida measured the drop across 12 models; prohibitions buried in long context were followed 73 percent of the time at turn 5 but only 33 percent by turn 16. Bigger context windows didn't fix it. What matters is where the constraint sits and how much surrounds it.

Cost is the third issue. Every call to the model reprocesses the entire context window. A 50,000-token context costs the same per call whether the last 500 tokens are new or the whole thing is.

The Gamage study tested compliance at multiple turn depths between turns 5 and 16 by setting a prohibition in the first system prompt and then running a controlled conversation in which intervening turns added context on top of it. The researchers tested two kinds of instructions: omissions (things the agent should not do, such as "never reveal credentials") and commissions (things the agent should always do, such as "always cite sources"). The omissions decayed across turns while commissions held steady at 100 percent.

Figure 1. Commission constraints hold; omission constraints decay as conversation grows.

The asymmetry is the core issue. The model isn't getting worse overall. Prohibitions, specifically the rules that protect users, enforce policy, and prevent leaks, fade as conversation grows. A reliable memory design has to keep those rules close to the model's active instructions.
Once the context window is treated as temporary working memory, the next question is what kinds of information should exist outside it.

The Four Types of Agent Memory

Cognitive Architectures for Language Agents (CoALA), a 2023 framework from Princeton, provides a useful taxonomy widely referenced in modern agent-memory discussions. It identifies four types of agent memory.
Working memory is the active context window. It holds the current task, recent tool results, the ongoing conversation, and the current task state. It gets cleared at session end.

Episodic memory is a record of specific past events what happened in previous sessions, what tools were called and what they returned, what the user said three conversations ago. It has to be stored externally to survive session resets.

Semantic memory is structured factual knowledge: user preferences, domain facts, and entity relationships. It doesn't change often and lives externally.

Procedural memory is rules, workflows, and decision patterns, usually implemented as part of the system prompt or as retrievable tool definitions.

In practice, episodic and semantic memory live in the same external storage layer. The distinction matters for retrieval logic, not for storage infrastructure. That gives us the design boundary for the rest of the article: working memory stays in the prompt, while durable facts and past events move into external storage.

The Two-Layer Architecture

The design becomes much easier if you separate memory by lifespan. Information that needs to persist into a future session goes into persistent storage; anything that only matters for the current session stays in the context window.

The working memory layer (the context window) holds the current task and immediate user request, tool results summarized to relevant facts before injection rather than raw API responses, the current plan and task state, and hard constraints re-injected at the top of every system prompt call. Constraints do not live in conversation history. They sit at the top of every system prompt, so the model can't bury them.

The persistent memory layer (external storage) holds user preferences with exact values rather than summaries: "user prefers 2700K warm white lighting after 8 p.m." rather than "user has lighting preferences." It also holds hard constraints that must survive session boundaries, identity facts like timezone and project context, and task state that needs to survive a session reset.

Figure 2. The two-layer memory architecture. Working memory clears at the end of the session; persistent memory survives.

At the end of every session, run an extraction pass that writes stable preferences and facts to persistent storage and discards everything else. That extraction is the primary route for transferring information from working memory to persistent memory.

Before writing anything to that external layer, though, it's worth deciding what should never be stored in the first place.

Memory Safety: What Not to Persist

Persistent memory is sensitive and should be treated like any other store of user data, which means writing only the stable facts the agent will need later rather than everything the user says. Tag each memory by type preference, constraint, identity fact, task state — so retrieval logic and access controls can handle them differently.

Hard constraints in particular should not be mixed with casual preferences. A policy that says "never share medical records" is not the same as "user prefers email over Slack," and storing them in the same untagged bucket invites confusion later. Apply standard data hygiene: minimize PII, give sensitive fields a retention window and a deletion path, encrypt at rest, and put access controls in front of the store. Consent for what gets remembered should match the surface where the agent is deployed.

It’s important to also plan for stale and conflicting memories. A user who preferred email in session 1 might switch to Slack in session 5. If both records exist and there is no metadata to choose between them, retrieval can return contradictory facts, and the agent will pick one at random. Stamp every memory with a timestamp and a memory type, and when an extractor writes a new preference, supersede the older record rather than storing both. Retrieval logic can then prefer the most recent valid fact for any given type.

With those rules in place, the storage layer becomes easier to choose. The right backend depends on where the memory is allowed to live, how much retrieval volume you expect, and how much infrastructure the team already operates.

Building the Persistent Memory Layer

Three paths cover most cases.
Teams already running Postgres with pgvector, Actian Zen, or HCL Informix® can embed vector search directly in their existing database. If the memory layer fits comfortably within that system, there's no reason to add another one. One system to operate, one place to back up.

A dedicated vector store is the right fit when the memory layer must stay local or on-premises, when the workload calls for filter-heavy retrieval, or when the dataset will outgrow what a relational extension comfortably handles. This is the most explicit version of the architecture, since the vector store exists for one job: serving the memory layer.

For deduplication, multi-strategy retrieval, and managed extraction pipelines, use a purpose-built memory layer like Mem0 or Zep.

For this tutorial, we'll use the dedicated vector store path because it shows the memory pattern most clearly, and we'll use VectorAI DB as the implementation. It runs locally and on-premises with Python and JavaScript clients designed for agent workloads. The same pattern works with a relational vector extension or a managed memory platform; only the client changes.

The implementation has four moving parts: start the vector store, create a memory collection, write extracted facts at session close, and retrieve relevant facts at the next session start. The agent loop at the end ties them together.

Run the vector store
VectorAI DB runs as a Docker container. The image exposes gRPC on port 6574 and a local web UI on 6575:

docker pull actian/vectorai:latest
docker run -d --name vectorai \
  -v ./local_data:/var/lib/actian-vectorai \
  -p 6573-6575:6573-6575 \
  -e ACTIAN_VECTORAI_ACCEPT_EULA=YES \
  actian/vectorai:latest

Figure 3. VectorAI DB running locally in a Docker container.

Install the Python client:

pip install actian-vectorai-client sentence-transformers

Create the memory collection

The memory record carries the verbatim text in the payload's content field so retrieval returns exact values, not summaries. Agent ID, session ID, and timestamp let you filter retrieval by agent, conversation, or recency:

from actian_vectorai import VectorAIClient, VectorParams, Distance
from sentence_transformers import SentenceTransformer
EMBED_MODEL = "all-MiniLM-L6-v2"
EMBED_DIM = 384
COLLECTION = "agent_memory"
embedder = SentenceTransformer(EMBED_MODEL)

with VectorAIClient("localhost:6574") as client:
    client.collections.get_or_create(
        name=COLLECTION,
        vectors_config=VectorParams(size=EMBED_DIM, distance=Distance.Cosine),
    )

Figure 4. The agent_memory collection in the VectorAI DB local UI.

Write at session close

At the end of every session, extract the stable facts from the conversation and write them down. Each memory is one PointStruct: an integer ID, an embedding of the content, and a payload that carries the verbatim text plus the routing metadata.

from actian_vectorai import VectorAIClient, PointStruct
from datetime import datetime, timezone
import uuid

def write_memory(content: str, agent_id: str, session_id: str):
    """Persist one extracted memory at session close."""
    vector = embedder.encode(content).tolist()
    memory_id = uuid.uuid4().int >> 64  # 64-bit positive int

    with VectorAIClient("localhost:6574") as client:
        client.points.upsert(COLLECTION, points=[
            PointStruct(
                id=memory_id,
                vector=vector,
                payload={
                    "content": content,                      
                    "agent_id": agent_id,
                    "session_id": session_id,
                    "timestamp": datetime.now(timezone.utc).isoformat(),
                },
            ),
        ])
        client.vde.flush(COLLECTION)

Retrieve at session start

At the start of the session, retrieve memories that match the user's likely intent. Filter by agent so memories don't leak between agents that share the store:

from actian_vectorai import VectorAIClient, Field, FilterBuilder
def retrieve_memories(query: str, agent_id: str, k: int = 5):
    """Pull the k most relevant memories for this agent."""
    query_vector = embedder.encode(query).tolist()
    agent_filter = FilterBuilder().must(Field("agent_id").eq(agent_id)).build()

    with VectorAIClient("localhost:6574") as client:
        return client.points.search(
            COLLECTION,
            vector=query_vector,
            limit=k,
            with_payload=True,
            filter=agent_filter,
        ) or []

Figure 5. Semantic retrieval returning the most relevant memory by vector similarity, not keyword match.

The agent loop

A simple baseline is one read at session start and one write at session close. Longer sessions, agents that take consequential actions, or workflows that may exit unexpectedly, need checkpoint writes after important state changes. That might be after a confirmed preference, after a tool call that mutates external state, or before a risky action. The pattern below is a starting point that production systems typically extend.

def run_agent_session(user_id: str, agent_id: str, hard_constraints: list[str]):
    session_id = str(uuid.uuid4())

    # 1. Read at session start
    memories = retrieve_memories(query=f"context for user {user_id}", agent_id=agent_id)
    memory_text = "\n".join(m.payload["content"] for m in memories)

    # 2. Build the system prompt with constraints pinned at the top, every call
    system_prompt = (
        "CONSTRAINTS (always apply):\n"
        + "\n".join(f"- {c}" for c in hard_constraints)
        + "\n\nRELEVANT MEMORY:\n"
        + memory_text
    )

    # 3. Run the conversation. Re-prepend the constraint block on every LLM call,
    #    not just at session start. Summarize tool results before injecting them.
    conversation = run_conversation(system_prompt)

    # 4. Write at session close: extract stable facts and persist
    for fact in extract_stable_facts(conversation):
        write_memory(content=fact, agent_id=agent_id, session_id=session_id)

run_conversation and extract_stable_facts are placeholders for your agent runtime and your extraction logic. A reasonable implementation of extract_stable_facts uses the LLM with a focused prompt to identify durable preferences and facts from the transcript ("user mentioned they prefer email over Slack" graduates to persistent memory; "user said hi" does not). Keep the extractor narrow, because anything it writes will influence future sessions.

Figure 6. The agent loop: one read at session start, one write at session close.

Once the read and write hooks are in place, the next step is proving that the memory layer actually fixes the failure modes the article started with.

Testing That It Works

Four tests will tell you whether the architecture is ready for a pilot. Each one targets a specific failure mode and provides a specific fix if it doesn't pass.

Constraint adherence by turn depth. Set one constraint at turn 1, then run the agent through turn 15. Check compliance at turns 5, 10, and 15. Target: above 90 percent at all three. If it drops below 70 percent by turn 10, the constraint probably isn't being pinned correctly. Pull it from persistent memory at session start and prepend it to the system prompt on every LLM call, not just the first one.

Exact value accuracy. Store a specific preference with an exact value ("2700K warm white lighting after 8 p.m."). Retrieve it two sessions later without restating it. Check that the exact value comes back, not a paraphrase.

Fix: Store the verbatim string in the content field of the memory record. Semantic search retrieves what's in the content, so if the exact value is there, it comes back.

Token growth over time. Measure working-memory token count at the start and end of a 20-turn session. A flat line or gentle growth indicates healthy management. Exponential growth means tool results are being injected raw.

Fix: Add a summarization step between tool execution and context injection. Inject only the facts the agent needs to continue.

Cross-session recall. State a fact in session 1, close the agent, open session 2, ask a question that requires that fact. If the agent asks for the fact again, the session-close extraction isn't running.

Fix: Confirm the extraction function is being called at session end (including non-graceful exits) and that client.vde.flush() completes before the session terminates.

If all four pass, the memory layer is ready for a basic production pilot. Full production readiness also depends on access control, encryption, observability, retention policy, and failure handling, none of which are covered by these tests.

Figure 7. Cross-session recall in practice: identical retrieval scores after a full container restart confirm the memories persisted to disk and were not rebuilt.

Wrapping Up

Persistent agent memory starts with deciding what belongs in the prompt and what belongs in storage. The context window holds the current task, recent facts, and summarized tool outputs. Anything that must survive the session's hard constraints, exact user preferences, or task state lives in persistent storage and is retrieved deliberately.
That separation is what prevents the turn-15 failure from the opening. The agent no longer depends on an outdated instruction buried in the conversation history.
For deeper implementation detail, see the VectorAI DB Python SDK reference for collection state, payload indexes, and connection pooling, and the legal contract intelligence agent tutorial for a more advanced example.

Informix is a trademark of IBM Corporation in at least one jurisdiction and is used under license.