Beyond the Prompt: Building AI Agents with Memory and Context

The Memory Gap in Modern AI

You’ve seen the demos. An AI agent, given a simple prompt, can write code, analyze data, or draft an email. It’s impressive, but it’s also fleeting. Ask a follow-up question, give a slightly adjusted instruction, or return to the conversation an hour later, and you’re often starting from scratch. As one popular article this week put it: "Your agent can think. It can't remember."

This is the core limitation of the stateless, prompt-based interactions that dominate today's AI landscape. For AI to evolve from a clever parlor trick into a reliable, persistent collaborator, it needs memory. It needs context. It needs to know what it did five minutes ago and why.

This guide dives into the technical architecture of AI agents with memory. We'll move beyond simple API calls to explore how to build systems that learn, adapt, and maintain a coherent thread of interaction over time. This isn't just theory; we'll build a practical blueprint you can implement.

Why Stateless AI Falls Short

Most applications using Large Language Models (LLMs) like GPT-4 or Claude are stateless. Each API call is an isolated event.

# A typical, stateless interaction
response_1 = llm_client.chat_completion(messages=[
    {"role": "user", "content": "Summarize the key points of our project brief."}
])

# This call has no knowledge of the first.
response_2 = llm_client.chat_completion(messages=[
    {"role": "user", "content": "Now, draft an email based on that summary."} # "that summary" is undefined!
])

The model in response_2 has no inherent knowledge of response_1. To simulate continuity, developers are forced to cram the entire conversation history into each new prompt, leading to ballooning token counts, increased costs, and eventual context window limits.

A true agent with memory operates differently. It persists information between interactions, creating a contextual loop.

Architecting Memory: The Core Components

Building a mnemonic AI agent requires a shift in design. Think of it as a system with three key layers:

1. The LLM (The Processor): The reasoning engine.
2. The Memory Store (The Knowledge): A database for past interactions, facts, and decisions.
3. The Orchestrator (The Manager): The logic that decides what to store, how to retrieve it, and when to use it.

Here’s a conceptual diagram of the data flow:

User Input -> Orchestrator -> [Queries Memory Store] -> Builds Context-Aware Prompt -> LLM -> Generates Response -> Orchestrator -> [Updates Memory Store] -> Returns Response to User

Implementing a Simple Memory Store

Let's build a basic ConversationMemory class using Python and SQLite. This is a foundational step.

import sqlite3
from datetime import datetime
from typing import List, Dict, Any
import json

class ConversationMemory:
    def __init__(self, db_path=":memory:"):
        self.conn = sqlite3.connect(db_path, check_same_thread=False)
        self._create_tables()

    def _create_tables(self):
        """Creates tables for storing interactions and key facts."""
        cursor = self.conn.cursor()
        # Table for raw conversation turns
        cursor.execute('''
            CREATE TABLE IF NOT EXISTS interactions (
                id INTEGER PRIMARY KEY AUTOINCREMENT,
                timestamp TEXT NOT NULL,
                role TEXT NOT NULL, -- 'user', 'assistant', 'system'
                content TEXT NOT NULL,
                session_id TEXT
            )
        ''')
        # Table for extracted/important facts (semantic memory)
        cursor.execute('''
            CREATE TABLE IF NOT EXISTS facts (
                id INTEGER PRIMARY KEY AUTOINCREMENT,
                entity TEXT,
                attribute TEXT,
                value TEXT,
                source_interaction_id INTEGER,
                timestamp TEXT NOT NULL,
                FOREIGN KEY (source_interaction_id) REFERENCES interactions (id)
            )
        ''')
        self.conn.commit()

    def add_interaction(self, role: str, content: str, session_id: str = "default"):
        """Stores a single message from user or assistant."""
        cursor = self.conn.cursor()
        timestamp = datetime.utcnow().isoformat()
        cursor.execute('''
            INSERT INTO interactions (timestamp, role, content, session_id)
            VALUES (?, ?, ?, ?)
        ''', (timestamp, role, content, session_id))
        self.conn.commit()
        return cursor.lastrowid

    def get_recent_context(self, session_id: str = "default", limit: int = 10) -> List[Dict]:
        """Retrieves the most recent interactions for context window."""
        cursor = self.conn.cursor()
        cursor.execute('''
            SELECT role, content FROM interactions
            WHERE session_id = ?
            ORDER BY timestamp DESC
            LIMIT ?
        ''', (session_id, limit))
        rows = cursor.fetchall()
        # Return in chronological order
        context = [{"role": role, "content": content} for role, content in reversed(rows)]
        return context

    def add_fact(self, entity: str, attribute: str, value: str, source_id: int):
        """Stores a structured fact extracted from an interaction."""
        timestamp = datetime.utcnow().isoformat()
        cursor = self.conn.cursor()
        cursor.execute('''
            INSERT INTO facts (entity, attribute, value, source_interaction_id, timestamp)
            VALUES (?, ?, ?, ?, ?)
        ''', (entity, attribute, value, source_id, timestamp))
        self.conn.commit()

# Usage Example
memory = ConversationMemory("agent_memory.db")
user_input = "My name is Alex, and I'm building a Python web app."
interaction_id = memory.add_interaction("user", user_input)

# Simulate an LLM extracting a fact
memory.add_fact(entity="User", attribute="name", value="Alex", source_id=interaction_id)
memory.add_fact(entity="User Project", attribute="type", value="Python web app", source_id=interaction_id)

This gives us durable storage. Now, we need the brains to use it.

The Orchestrator: Making Memory Useful

The orchestrator's job is to dynamically construct the prompt sent to the LLM. A naive way is to just slap the last 10 messages into the prompt. A smarter way uses semantic search to find relevant past interactions, not just recent ones.

Let's enhance our orchestrator with a simple vector-based retrieval system. We'll use sentence-transformers to embed text and find similar past conversations.

# pip install sentence-transformers
from sentence_transformers import SentenceTransformer
import numpy as np

class SmartOrchestrator:
    def __init__(self, memory: ConversationMemory, embedding_model_name='all-MiniLM-L6-v2'):
        self.memory = memory
        self.embedder = SentenceTransformer(embedding_model_name)
        # Cache for interaction embeddings
        self._embedding_cache = {}

    def build_context_prompt(self, current_query: str, session_id: str = "default") -> List[Dict]:
        """Builds a prompt with relevant context from memory."""
        prompt_messages = []

        # 1. Get recent context for conversational flow
        recent_context = self.memory.get_recent_context(session_id=session_id, limit=5)
        prompt_messages.extend(recent_context)

        # 2. Find semantically relevant past interactions (beyond recent)
        relevant_interactions = self._find_semantically_relevant(current_query, session_id, top_k=3)
        for role, content in relevant_interactions:
            # Avoid duplicating messages already in recent context
            if not any(msg['content'] == content for msg in prompt_messages):
                prompt_messages.append({"role": role, "content": f"[Relevant past context] {content}"})

        # 3. Query for relevant structured facts
        facts_context = self._get_relevant_facts(current_query)
        if facts_context:
            system_fact_msg = {"role": "system", "content": f"Known facts: {facts_context}"}
            # Insert at the beginning after any system instructions
            prompt_messages.insert(0, system_fact_msg)

        # 4. Add the current query
        prompt_messages.append({"role": "user", "content": current_query})
        return prompt_messages

    def _find_semantically_relevant(self, query: str, session_id: str, top_k: int=3):
        """Uses vector similarity to find related past interactions."""
        # In a full implementation, you would store and index embeddings in the database.
        # This is a simplified in-memory version for illustration.
        query_embedding = self.embedder.encode(query)
        # ... (Code to fetch stored interaction embeddings and compute cosine similarity) ...
        # Returns a list of (role, content) tuples
        return []  # Placeholder

    def _get_relevant_facts(self, query: str) -> str:
        """Retrieves facts potentially related to the query."""
        cursor = self.memory.conn.cursor()
        # Simple keyword matching for illustration. Use embeddings for production.
        cursor.execute('''
            SELECT entity, attribute, value FROM facts
            WHERE entity LIKE ? OR attribute LIKE ? OR value LIKE ?
            LIMIT 5
        ''', (f'%{query}%', f'%{query}%', f'%{query}%'))
        rows = cursor.fetchall()
        return "; ".join([f"{entity}'s {attribute} is {value}" for entity, attribute, value in rows])

# Putting it all together
memory = ConversationMemory()
orchestrator = SmartOrchestrator(memory)

# Simulate a conversation
memory.add_interaction("user", "I prefer dark mode for IDEs.")
memory.add_interaction("assistant", "Noted. Dark mode is easier on the eyes for many developers.")

# Later, a new query
new_query = "What were my preferences for the development environment?"
contextual_prompt = orchestrator.build_context_prompt(new_query)
# contextual_prompt now contains the past conversation about dark mode,
# allowing the LLM to give a personalized response.
print(contextual_prompt)

Advanced Patterns: Reflection and Summarization

To prevent memory from growing unbounded, sophisticated agents use reflection. Periodically, an agent can analyze its recent interactions and summarize them into higher-order insights or compress them, moving details from short-term conversational memory to long-term semantic memory.

def reflective_summarization(memory: ConversationMemory, session_id: str):
    """Uses an LLM to summarize recent interactions and store key takeaways."""
    recent_chat = memory.get_recent_context(session_id=session_id, limit=20)
    # Format chat for summarization
    chat_text = "\n".join([f"{msg['role']}: {msg['content']}" for msg in recent_chat])

    # Prompt an LLM to summarize key decisions, facts, and user preferences
    summary_prompt = f"""
    Analyze the following conversation and extract persistent facts, user preferences, and key decisions.
    Format the output as a concise list of JSON objects with 'entity', 'attribute', and 'value' keys.

    Conversation:
    {chat_text}
    """
    # ... Call LLM with summary_prompt ...
    # Parse LLM response and store as facts using memory.add_fact()
    # Optionally, clear or archive the old raw interactions to save space

The Road Ahead: From Memory to True Agency

Implementing memory transforms your AI from a tool into a nascent collaborator. The next steps on this path involve:

• Tool Integration: Allowing the agent to take actions (send emails, run code, query APIs) and remember the outcomes.
• Autonomous Goal Management: The agent breaking down a high-level user goal ("Build a marketing plan") into sub-tasks, executing them, and remembering progress.
• Learning from Feedback: Explicitly storing user corrections ("No, not like that, like this.") to avoid repeating mistakes.

Start Building Your Mnemonic Agent

The gap between "thinking" and "remembering" is the most exciting frontier in practical AI right now. You don't need a billion-parameter model to start; you need a thoughtful architecture.

Your Call to Action: Take one of your existing LLM projects. Integrate a simple memory store using SQLite or even a JSON file. Start by just persisting the conversation. Then, add a function to pull in the last three messages automatically. You'll immediately feel the difference in interaction quality.

Share what you build. The community is rapidly iterating on these patterns, and your experiments will contribute to the collective understanding of how to create truly useful AI agents. The era of the forgetful AI is ending. Let's build what comes next.