DEV Community

BAOFUFAN
BAOFUFAN

Posted on

How We Cut AI Memory Regression Testing from 30 Minutes to 2 Minutes

#python #programming

At 2 a.m., a QA colleague sent a screenshot in our group chat: half of the regression test cases were failing again. Not because of code bugs, but because manually verifying AI conversation memory storage involved too many steps and was extremely error-prone. Before every release, we had to open a browser, act as a user, have a 5-round conversation with the LLM, log out, log back in, and check if the memory persisted—verifying both Redis and the database. Each run took 30 minutes per person, and we could never be sure there wasn’t a mistake. I decided to get rid of this manual clicking once and for all and let Playwright + pytest take over.

Breaking Down the Problem: Why Is Memory Storage Testing So Hard for LLM Applications?

Our AI application maintains a conversation memory for each user, stored in Redis under the key chat:memory:{user_id}, and asynchronously persisted to the chat_memory table in PostgreSQL. End-to-end tests need to verify:

  • After multiple dialogue turns, new context is appended to memory.
  • Logging out and logging back in restores memory correctly.
  • Memory TTL (time-to-live) policy takes effect.
  • Database and Redis remain consistent.

Conventional unit tests can only mock the storage layer; they can’t cover real browser interactions or the timing of state transitions on the backend. API integration tests can verify memory writes, but they can’t validate that the frontend correctly displays the memory content. We once had a bug where the backend stored the memory, but the frontend never showed it because of a field name typo. Users only saw a blank conversation history. Manual testing caught it—all unit tests were green.

To be truly confident, we had to simulate real user actions: open browser → log in → multi-turn dialogue → close page → reopen → check memory state. Doing this manually is tedious, time-consuming, and not reproducible. Selenium’s wait strategies were flaky—scripts would randomly fail due to AJAX delays.

Solution: Playwright + pytest + Direct Storage Checks for Three-Way Validation

We didn’t hesitate long on the tech choices:

  • Playwright: Faster than Selenium, built-in auto-wait, highly readable generated code, and browser context isolation—so one test file can simulate multiple independent user sessions.
  • pytest: Our team works mainly with Python. pytest’s fixture system elegantly manages browser instances and test data, all injected from conftest.py.
  • Direct Redis/DB queries: We weren’t satisfied with asserting just the page DOM—that only proves the frontend shows some text, not that the memory actually persisted. We introduced redis-py and sqlalchemy directly in test cases to query storage, forming a three-way check: page display + Redis content + DB record. Any mismatch in any layer would be caught immediately.

Why not Cypress? Our backend is a Python stack, and test scripts need to interact with Redis and the database. Using Python feels natural and lets us reuse existing ORM models. Plus, Playwright handles multi-tab and multi-user concurrency scenarios better—crucial for simulating logout/login flows.

Overall architecture: pytest collects test cases → conftest provides the page fixture (auto-creating browser contexts) → each test uses the page object to interact with the chat UI → at the end of the test, it asserts memory data via redis_client and the database session. One test run covers both UI and backend state.

Core Implementation: One Fixture to Rule Them All

This code handles browser lifecycle management and Redis connection reuse.

# tests/conftest.py
import pytest
from playwright.sync_api import sync_playwright
import redis
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

@pytest.fixture(scope="session")
def browser():
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)  # CI环境必须headless
        yield browser
        browser.close()

@pytest.fixture
def page(browser):
    context = browser.new_context()  # 隔离会话，避免Cookie污染
    page = context.new_page()
    yield page
    context.close()

@pytest.fixture(scope="session")
def redis_client():
    r = redis.Redis(host='localhost', port=6379, decode_responses=True)
    yield r
    r.close()

@pytest.fixture(scope="session")
def db_session():
    engine = create_engine('postgresql://user:pass@localhost/testdb')
    Session = sessionmaker(bind=engine)
    session = Session()
    yield session
    session.close()
Enter fullscreen mode Exit fullscreen mode

The page fixture is per test; just create a new page for the logout test, and you get a fresh browser environment, perfectly simulating a user clearing local state.

This code simulates multi-turn dialogue and verifies that memory is correctly written to Redis.

# tests/test_memory_storage.py
import pytest
import time

def test_memory_recorded_after_dialogue(page, redis_client):
    # 1. 登录（假设应用部署在本地8000端口）
    page.goto("http://localhost:8000/login")
    page.fill("input[name=username]", "testuser")
    page.fill("input[name=password]", "password123")
    page.click("button:has-text('登录')")
    page.wait_for_url("**/chat")  # 等待跳转到聊天页

    # 2. 多轮对话
Enter fullscreen mode Exit fullscreen mode

In tests/test_memory_storage.py, we simulate a user logging in, sending three messages, and then assert the AI’s memory content. Note: Playwright’s page.fill and page.click already wait for elements to be actionable, but memory writes are asynchronous, so we must explicitly wait for the Redis key to meet a condition.

Top comments (0)