From 3 Hours to 5 Minutes: Automating AI Chatbot Memory Regression with Pytest & Docker

It was 1 AM when a Slack alert jolted me awake: a user complained that “it suddenly lost its memory”—I had just told the AI I'm from Hangzhou, but in the next turn when asked “Where is my hometown?”, the bot replied deadpan, “I don't know.” Checking the latest release, sure enough the memory storage module had changed its serialization logic, quietly breaking the path for loading conversation history.

A manual regression pass took 3 hours: set up Redis and Postgres environments separately, simulate multi-turn conversations, switch users, check memory recall… After that night I decided: this grunt work had to be automated away, once and for all.

Where Exactly Is the Pain?

Memory storage in an AI chatbot isn't as simple as saving a single key-value pair. It involves:

• Multiple storage backends: Redis for short‑term, Postgres for long‑term, sometimes a hybrid architecture
• Memory lifecycle: reads/writes, expiration, merging, rollback
• Context and user isolation: threading memories across multiple sessions of the same user, while strictly preventing cross‑user contamination
• Sensitivity to release changes: a single change to serialization format, schema, or TTL logic can silently drop memories

The typical approach is to mock the storage layer in unit tests, but mocks can't expose the real serialization/deserialization pitfalls—many production incidents were cases where mocks passed with flying colors, only to fail miserably in production. A manual regression pass went like: spin up environments → fabricate conversation data → deploy the change → verify memories → fabricate more data → verify again. Half an hour was the bare minimum; a full‑scenario regression could easily take 3 hours and still miss bugs.

Design: Turn Real Middleware into Disposable Test Environments

The core idea: use Docker containers to provide real Redis/Postgres, drive tests with Pytest parametrization, and turn memory logic into repeatable regression test cases.

Why not other approaches?

• Why not fakeredis or testcontainers-python with an in-memory fake Redis? Because serialization/deserialization, Lua script behavior under cluster mode, and other nuances can't be replicated by a fake library—that only gives you false confidence.
• Why not a shared remote environment integrated with CI? Multiple tests running in parallel would pollute each other, plus it's expensive and slow.
• What about Docker + the pytest-docker plugin? The plugin's session scope turned out to be tricky (more on that later). Ultimately I chose pytest fixture + manual lifecycle control with docker-compose, which proved flexible and stable.

Architecture: docker-compose.yml defines the Redis and Postgres services; a module-scoped fixture in conftest.py handles docker-compose up and tears it down after tests; test cases use parametrize to cover different storage backends, conversation turns, and data sizes, verifying memory read/write, isolation, and migration at a click.

Core Implementation: Building the Regression Step by Step

1. Use docker-compose to define disposable real middleware

This config solves the problem of differing databases across environments—dev, CI, and local all use the same image versions.

# docker-compose.yml
version: "3.8"
services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379"
    # No fixed external port to avoid conflicts
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_USER: test
      POSTGRES_PASSWORD: test
      POSTGRES_DB: memory_test
    ports:
      - "5432"

2. Use a fixture in conftest.py to manage container lifecycle and inject dynamic connection info

This code eliminates the pain of manually starting services every time you run tests, and dynamically assigns ports to avoid conflicts during parallel runs.

# conftest.py
import pytest
import docker
import time
import os

# Use module scope to balance speed and isolation
@pytest.fixture(scope="module")
def docker_services():
    client = docker.from_env()
    # Start with docker-compose, isolated project name prevents clashes
    project_name = f"memtest_{os.getpid()}"
    compose_file = os.path.join(os.path.dirname(__file__), "docker-compose.yml")
    # Use the docker-compose CLI (or Compose API, but CLI is more universal)
    os.system(f"docker-compose -p {project_name} -f {compose_file} up -d")
    # Wait for Redis/Postgres to be ready (health check polling)
    redis_container = None
    pg_container = None
    for c in client.containers.list(filters={"name": project_name}):
        if "redis" in c.name:
            redis_container = c
        elif "postgres" in c.name:
            pg_container = c

    # Poll until Redis is ready
    for _ in range(30):
        exit_code, _ = redis_container.exec_run("redis-cli ping")
        if b"PONG" in exit_code:
            break
        time.sleep(0.5)
    else:
        raise RuntimeError("Redis did not start")

    # Wait for Postgres to accept connections
    for _ in range(30):
        exit_code, _ = pg_container.exec_run("pg_isready -U test")
        if b"accepting" in exit_code:
            break
        time.sleep(0.5)
    else:
        raise RuntimeError("Postgres did not start")

    # 拿到动态端口
    redis_port = int(redis_container.attrs["NetworkSettings"]["Ports"]["6379/tcp"][0]["Ho