Implementing Conversation Memory in AI Apps: Short-Term, Long-Term, and Context Compression

#typescript #ai #webdev #nextjs

Most AI chat applications lose context after each session. The user has to re-explain their project, their constraints, and their preferences every time. That's a bad product.

Here's how to implement persistent conversation memory that makes your AI app feel genuinely intelligent across sessions.

Two Types of Memory

Short-term memory: The current conversation history. Claude can reference anything said earlier in the same session.

Long-term memory: Facts, preferences, and context that persist across sessions. This is what most apps are missing.

Storing Conversation History

Start with the basics -- persist messages to the database.

// prisma/schema.prisma
model Conversation {
  id        String    @id @default(cuid())
  userId    String
  title     String?
  createdAt DateTime  @default(now())
  updatedAt DateTime  @updatedAt

  user     User      @relation(fields: [userId], references: [id], onDelete: Cascade)
  messages Message[]
}

model Message {
  id             String   @id @default(cuid())
  conversationId String
  role           String   // "user" | "assistant" | "system"
  content        String   @db.Text
  inputTokens    Int      @default(0)
  outputTokens   Int      @default(0)
  createdAt      DateTime @default(now())

  conversation Conversation @relation(fields: [conversationId], references: [id], onDelete: Cascade)
}

// src/app/api/chat/route.ts
import { NextRequest, NextResponse } from "next/server"
import { auth } from "@/lib/auth"
import Anthropic from "@anthropic-ai/sdk"
import { db } from "@/lib/db"

const claude = new Anthropic()

export async function POST(req: NextRequest) {
  const session = await auth()
  if (!session?.user?.id) {
    return NextResponse.json({ error: "Unauthorized" }, { status: 401 })
  }

  const { conversationId, content } = await req.json()

  // Load conversation history
  const history = conversationId
    ? await db.message.findMany({
        where: { conversationId },
        orderBy: { createdAt: "asc" },
        select: { role: true, content: true },
      })
    : []

  // Create conversation if new
  const conversation = conversationId
    ? { id: conversationId }
    : await db.conversation.create({
        data: { userId: session.user.id, title: content.slice(0, 50) }
      })

  // Save user message
  await db.message.create({
    data: {
      conversationId: conversation.id,
      role: "user",
      content,
    }
  })

  // Build messages array with history
  const messages = [
    ...history.map(m => ({ role: m.role as "user" | "assistant", content: m.content })),
    { role: "user" as const, content },
  ]

  const response = await claude.messages.create({
    model: "claude-sonnet-4-6",
    max_tokens: 1024,
    messages,
  })

  const assistantContent = response.content[0].type === "text"
    ? response.content[0].text
    : ""

  // Save assistant response
  await db.message.create({
    data: {
      conversationId: conversation.id,
      role: "assistant",
      content: assistantContent,
      inputTokens: response.usage.input_tokens,
      outputTokens: response.usage.output_tokens,
    }
  })

  return NextResponse.json({
    content: assistantContent,
    conversationId: conversation.id,
  })
}

The Context Window Problem

Long conversations eventually exceed Claude's context window. You have two options:

Option 1: Sliding Window

Keep only the most recent N messages:

const MAX_HISTORY_MESSAGES = 20

const history = await db.message.findMany({
  where: { conversationId },
  orderBy: { createdAt: "desc" },
  take: MAX_HISTORY_MESSAGES,
  select: { role: true, content: true },
})

// Reverse to chronological order
const messages = history.reverse()

Simple, predictable. Loses context from early in the conversation.

Option 2: Summarization

When history gets long, summarize the old portion:

async function getCompressedHistory(conversationId: string) {
  const allMessages = await db.message.findMany({
    where: { conversationId },
    orderBy: { createdAt: "asc" },
  })

  if (allMessages.length <= 20) {
    return allMessages.map(m => ({ role: m.role, content: m.content }))
  }

  // Summarize everything except the last 10 messages
  const toSummarize = allMessages.slice(0, -10)
  const recent = allMessages.slice(-10)

  const summary = await claude.messages.create({
    model: "claude-haiku-4-5",  // Cheap model for summarization
    max_tokens: 512,
    messages: [
      {
        role: "user",
        content: `Summarize this conversation in 3-5 sentences, focusing on key decisions, facts established, and context that would be important to remember:

${toSummarize.map(m => `${m.role}: ${m.content}`).join("

")}`
      }
    ]
  })

  const summaryText = summary.content[0].type === "text"
    ? summary.content[0].text
    : ""

  return [
    { role: "system", content: `Previous conversation summary: ${summaryText}` },
    ...recent.map(m => ({ role: m.role, content: m.content })),
  ]
}

More complex but preserves important context from the full history.

Long-Term Memory: Extracting and Storing Facts

For real persistence across sessions, extract important facts from conversations and store them separately.

model UserMemory {
  id        String   @id @default(cuid())
  userId    String
  key       String   // e.g., "preferred_stack", "current_project"
  value     String   @db.Text
  source    String?  // Which conversation this came from
  createdAt DateTime @default(now())
  updatedAt DateTime @updatedAt

  user User @relation(fields: [userId], references: [id], onDelete: Cascade)

  @@unique([userId, key])
}

After each conversation turn, extract and store facts:

async function extractAndStoreMemory(userId: string, content: string) {
  // Use a cheap, fast model for extraction
  const extraction = await claude.messages.create({
    model: "claude-haiku-4-5",
    max_tokens: 256,
    system: "Extract any factual information about the user's preferences, project, or constraints from this message. Return as JSON: {facts: [{key: string, value: string}]}. Return empty facts array if nothing notable.",
    messages: [{ role: "user", content }],
  })

  const text = extraction.content[0].type === "text" ? extraction.content[0].text : "{}"

  try {
    const { facts } = JSON.parse(text)
    for (const fact of facts) {
      await db.userMemory.upsert({
        where: { userId_key: { userId, key: fact.key } },
        create: { userId, key: fact.key, value: fact.value },
        update: { value: fact.value },
      })
    }
  } catch {
    // Extraction failed -- not critical
  }
}

Then inject stored memory into each new conversation:

async function buildSystemPrompt(userId: string): Promise<string> {
  const memories = await db.userMemory.findMany({
    where: { userId },
    select: { key: true, value: true },
  })

  if (memories.length === 0) return "You are a helpful assistant."

  const memoryContext = memories
    .map(m => `${m.key}: ${m.value}`)
    .join("
")

  return `You are a helpful assistant. Here's what you know about this user:
${memoryContext}

Use this context to give personalized, relevant responses.`
}

Frontend: Conversation List

"use client"

export function ConversationSidebar({ userId }: { userId: string }) {
  const [conversations, setConversations] = useState([])
  const [activeId, setActiveId] = useState<string | null>(null)

  useEffect(() => {
    fetch("/api/conversations")
      .then(r => r.json())
      .then(setConversations)
  }, [])

  return (
    <aside className="w-64 border-r p-4">
      <button
        onClick={() => setActiveId(null)}
        className="w-full btn-secondary mb-4"
      >
        New conversation
      </button>
      <div className="space-y-1">
        {conversations.map(conv => (
          <button
            key={conv.id}
            onClick={() => setActiveId(conv.id)}
            className={cn(
              "w-full text-left px-3 py-2 rounded text-sm truncate",
              activeId === conv.id ? "bg-accent" : "hover:bg-muted"
            )}
          >
            {conv.title || "Untitled"}
          </button>
        ))}
      </div>
    </aside>
  )
}