200x Speedup: LLM Memory Regression Testing from 3 Days to 15 Minutes

At 1 a.m., the product manager spammed the group chat with three messages: "Users report the model has amnesia—it just recommended a steakhouse right after being told the user is a vegetarian." Investigation revealed that a serialization bug had slipped in during an update to the memory storage service. The real gut punch? This bug should have been caught in regression—but nobody had run the full manual regression suite in three weeks, because completing a single pass through conversation memory scenarios takes three days.

That's the testing nightmare with LLM memory storage: long scenarios, tons of state, complex assertions, and a manual regression cost so high you're tempted to quit. By automating the entire flow with Playwright + pytest, I shrank regression time from 3 days down to 15 minutes, covering all 135 scenarios. Here's the full breakdown of the key code and the pitfalls—in one shot.

Breaking Down the Problem

The core requirement for LLM memory storage is straightforward: across multiple conversation turns, the system must remember user preferences, historical facts, and session context, then accurately recall them later. Testing such a system calls for at least these steps:

1. Start an initial conversation and inject information ("My name is Zhang San, I’m allergic to seafood").
2. Open a new session (or wait some time) and verify that the memory is correctly recalled with a different question.
3. At the same time, check UI interactions—certain products display "memory cards" in the frontend that users can edit or delete.

Pure API testing (calling endpoints directly via requests) can handle data‑level validation, but it misses frontend logic: whether memory cards render properly, whether the delete action triggers a confirmation dialog, or even whether memory storage requests are fired from the frontend SDK. With manual testing, clicking through all these steps takes three days per round, and manual operations make it painfully easy to overlook edge cases like memory expiration time, concurrent writes, or cross‑session context pollution. The team wanted to automate, but Selenium‑based attempts were too brittle—flaky element waits drove the authors into despair.

Solution Architecture

The final stack: Playwright + pytest + allure, running entirely in Docker containers on CI.

I ruled out Selenium for simple reasons: it’s slow, its API is dated, and its auto‑wait mechanism is weak. Playwright brings native multi‑browser support, network interception, and the trace viewer—an absolute game‑changer for debugging asynchronous interactions like memory recall.

A few guiding principles of the test architecture:

• Use pytest fixtures to manage browser instances and test‑user lifecycles — each test function receives a logged‑in Page object in complete isolation.
• Encapsulate common conversational steps in conftest.py: start a conversation, send a message, wait for a reply, verify a memory card appears/disappears.
• Dual‑layer verification: assert memory card content via the UI, and intercept network requests to verify backend API request/response bodies, ensuring data consistency.
• Parallel execution: with pytest-xdist, 135 scenarios run on 4 workers in parallel, finishing the entire regression in 15 minutes.

The real value isn’t just “it runs faster”—it’s that we turned memory storage testing from a one‑time mega‑regression into something that runs on every commit. No more worrying “did anyone run regression?”

Core Implementation

1. Decoupling browser and user state with fixtures

This snippet ensures each test case gets a fully isolated, already‑logged‑in page, preventing cookie and localStorage pollution between tests while reusing the browser instance to keep startup overhead low.

# conftest.py
import pytest
from playwright.sync_api import sync_playwright, Page, Browser

@pytest.fixture(scope="session")
def browser():
    # 整个测试会话只启动一次浏览器
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True, args=["--no-sandbox"])
        yield browser
        browser.close()

@pytest.fixture()
def logged_page(browser: Browser, request) -> Page:
    # 每个测试函数一个独立context，隔离存储状态
    context = browser.new_context(
        storage_state=None,  # 不共享登录态
        locale="zh-CN"
    )
    page = context.new_page()
    # 模拟登录，这里用自定义函数，可根据自己系统替换
    login(page, test_user=request.param.get("user", "default"))
    yield page
    context.close()

Why new_context instead of new_page? Because a Playwright browser context works like an incognito window—it has its own storage, which is exactly the isolation level we need. request.param lets tests dynamically inject different‑role users, checking that “memories aren’t leaked across users.”

2. Writing test cases for memory injection and recall verification

This code walks through the full business loop: “state information → leave session → new session recall → assert memory exists → delete memory via UI → assert it’s gone.”

# test_memory_retention.py
import pytest
from playwright.sync_api import expect

@pytest.mark.parametrize("logged_page", [{"user": "user_zhangsan"}], indirect=True)
def test_allergy_memory_recall_after_session_reset(logged_page):
    page = logged_page

    # Step 1: 第一段对话，注入记忆
    page.goto("/chat/new")
    page.fill("textarea#prompt", "我叫张三，我对海鲜过敏，请记住")
    page.click("button#send")
    # 等待助手回复中出现确认关键词，避免硬编码time.sleep
    page.wait_for_selector("text=已经记住了", state="visible", timeout=15000)

    # Step 2: 关闭当前会话，模拟会话结束
    page.click("button#end-session")
    page.wait_for_selector("text=会话已结束", state="visible")

    # Step 3: 开启全新会话，验证记忆是否跨会话保留
    page.goto("/chat/new")
    page.fill("textarea#prompt", "帮我推荐一道菜")
    page.click("button#send")

    # 核心断言：助手的回复应该避开海鲜
    assis

That last line is deliberately kept as‑is—it represents where you’d grab the assistant’s reply and assert that it contains no seafood recommendations. The full assertion can be inserted depending on your specific UI. The key point: with this pattern, every commit can now verify memory retention in just 15 minutes, catching serialization bugs, UI inconsistencies, and cross‑session leaks before they ever reach your users.