Building a production-ready AI chatbot with memory and context

Most chatbot tutorials end after the first response. Production chatbots need to remember what was said three turns ago, survive restarts, and stay within token budgets — that's where the real work is. Skip that layer and you'll be rewriting core logic under pressure at 2am.

The statefulness problem

A language model has no memory between API calls. Every request starts from a blank slate. If you want the bot to remember the user's name from turn one, you have to send that history yourself in every subsequent request.

That works fine in demos. In production, you'll hit the model's context limit fast — and you'll pay for every token you resend. The solution is a memory layer between your application and the model: it stores conversation history, decides what to include in each request, and handles gracefully the case where history grows too large.

Three components make this work:

1. Session store — persists message history with a TTL (Redis is a natural fit)
2. Context builder — selects which messages fit within the token budget
3. API wrapper — calls the model and appends responses to history

Implementing the session store

I'll use Redis because it handles TTL natively, which matters for chatbot sessions: you want idle conversations to expire automatically rather than accumulating forever. tiktoken handles token counting server-side.

import json
import redis
import tiktoken

r = redis.Redis(host="localhost", port=6379, db=0)
enc = tiktoken.get_encoding("cl100k_base")

SESSION_TTL = 1800  # 30 minutes

def load_history(session_id: str) -> list[dict]:
    raw = r.get(f"chat:{session_id}")
    if not raw:
        return []
    return json.loads(raw)

def save_history(session_id: str, messages: list[dict]) -> None:
    r.set(f"chat:{session_id}", json.dumps(messages), ex=SESSION_TTL)

def count_tokens(messages: list[dict]) -> int:
    total = 0
    for msg in messages:
        total += 4  # per-message overhead (role, separators)
        total += len(enc.encode(msg.get("content", "")))
    return total + 2  # reply priming tokens

One detail worth calling out: the +4 per message and +2 at the end are not arbitrary. They match the actual overhead the API charges for message formatting. If you skip this, your token budget will drift and you'll occasionally hit limit errors on the next turn.

Building the context window manager

This is where most tutorials skip ahead and production systems break. If you blindly append messages to each request, you'll eventually exceed the model's context limit and surface an error at the worst possible moment.

The strategy here: keep a hard token budget, always include the system prompt, and trim from the oldest end first. The most recent turns are almost always more relevant than what was said ten exchanges ago.

def build_context(
    system_prompt: str,
    history: list[dict],
    token_budget: int = 3800,
) -> list[dict]:
    system = [{"role": "system", "content": system_prompt}]
    system_tokens = count_tokens(system)
    budget = token_budget - system_tokens

    selected = []
    for msg in reversed(history):
        msg_tokens = count_tokens([msg])
        if budget - msg_tokens < 0:
            break
        selected.insert(0, msg)
        budget -= msg_tokens

    return system + selected

Set token_budget to about 80% of the model's context limit, leaving headroom for the response. For a model with a 16k context, token_budget=12800 with max_tokens=2048 on the response leaves a safe margin.

One edge case to plan for: if a user pastes a large block of text in a single message, that message alone may exceed the budget. The loop above skips it cleanly, but the user's input will be silently dropped. Add a pre-check that rejects user messages over a configurable character limit and returns a friendly error.

Wiring it together

import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["LLM_API_KEY"])

SYSTEM_PROMPT = """You are a support assistant for a B2B SaaS product.
Answer questions accurately and concisely. If you don't know the answer, say so."""

def chat(session_id: str, user_message: str) -> str:
    history = load_history(session_id)
    history.append({"role": "user", "content": user_message})

    context = build_context(SYSTEM_PROMPT, history)

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=context,
        temperature=0.2,
        max_tokens=512,
    )

    assistant_reply = response.choices[0].message.content
    history.append({"role": "assistant", "content": assistant_reply})
    save_history(session_id, history)

    return assistant_reply

Usage:

sid = "user-42-session-001"
print(chat(sid, "What are your pricing tiers?"))
print(chat(sid, "Which plan includes SSO?"))  # second call sees the first exchange

The second call automatically includes the first exchange. If the user returns after the 30-minute TTL, the session is expired and the conversation starts fresh — no stale context carrying over.

What you'll need before going live

The code above handles the core logic. Before exposing it to real users, a few additions are non-negotiable:

Idempotency: network failures cause clients to retry. If you call the model twice for the same user message, you'll charge twice and return inconsistent replies. Store a request_id (generated client-side) with each exchange and return the cached reply if you've seen that ID before.

Cost tracking per session: log response.usage.prompt_tokens and response.usage.completion_tokens on every call. In practice, you'll find a small fraction of users consuming a disproportionate share of context — usually because they paste large documents mid-conversation. Knowing this before your first billing surprise is useful.

Summarization fallback: when history exceeds the token budget and you're trimming, you lose early context. For most support bots this is fine. For assistants that need to remember facts across long conversations — the user's name, their plan tier, decisions made in turn one — consider a separate summarization step: when history crosses a threshold, summarize the oldest N turns into a single context block and store it alongside the raw history.

Security: chatbot APIs are a common target for prompt injection. A user can attempt to override the system prompt by embedding instructions in their message. This doesn't require a complex mitigation strategy, but it does require thinking about what your system prompt grants and what the model will refuse to do. Reviewing a security hardening checklist before launch is a practical starting point.

The takeaway

Memory in a chatbot is infrastructure, not a feature. Get the session store and context builder right early; retrofitting them after you have users is painful. The token budget logic is the most important piece — models have hard limits, and exceeding them breaks conversations in ways that are hard to diagnose.

The pattern above (Redis + token-aware trimming + latest-first context) covers most production cases. If you need the bot to remember facts across weeks of conversation rather than a single session, semantic memory retrieval becomes relevant — but that's a separate layer on top of this one, not a replacement.

---

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.