From 2-Hour Manual Regression to 8 Minutes: Doubling Accuracy in LLM Memory Testing

#python #programming

It’s 2 a.m. and I’m jolted awake by WeCom (WeChat Work) alerts. I check the logs—yikes. Our customer service Agent’s memory storage module silently crashed in the middle of the night. All conversation contexts were gone. Every user was greeted with "Hello, I'm new here, how can I help you?" on repeat. The client exploded, and the PM fired off three existential questions in the group chat: "Don't we have automated tests?" "Did you run the regression?" "Why didn't we catch this?"

I stared at the "manual click-through" test cases I'd written a week ago, my face greener than after three cups of Americano — those tests didn’t cover memory persistence at all. Every release, we’d manually switch browsers, open multiple tabs, simulate cross-session conversations... one round took at least 2 hours, and we’d still miss cases.

This couldn't happen again. I decided to build an end-to-end flow with Playwright. This article is my post-mortem after smoothing out all the bumps.

Breaking Down the Problem: What Makes Testing LLM Memory So Hard

LLM "memory" isn’t simple CRUD. Take our customer service Agent: User A mentioned their name is Zhang San, they’re in Shenzhen, and their budget is 5,000 RMB — this info gets extracted into structured context and stored in a vector DB or Redis. Next time A asks, "What can I buy with my previous budget?", the Agent needs to fish "budget:5000" out of memory, combine it with the new message, re-compose the prompt, and stream it to the LLM.

The core pain point is the cross-session context chain:

Memory writing: Does the Agent correctly extract and store the user profile during the first conversation?
Memory retrieval: In a second conversation (new session or after reconnect), can the Agent recall the previously stored structured memory?
Memory fusion: Old memory + new message → is the LLM’s final answer contextually coherent?

How’s manual testing done? Open two incognito windows, simulate two conversations, and eyeball whether the responses "remember" the earlier settings. It’s slow, subjective, and a regression run feels like cultivating immortality. Meanwhile, off-the-shelf API tools only test individual endpoints — you can verify storage, queries, and responses, but you can’t confirm whether the natural language answer from the LLM actually used that memory.

Solution Design: Why Playwright + Full Browser Pipeline

Three options on the table:

Approach	Disadvantages
Pure API testing (requests directly to backend APIs)	Can’t test the data flow at the frontend WebSocket layer, let alone verify the semantic correctness of LLM responses.
Selenium + custom assertions	WebSocket support is a pain, weak multi-tab capabilities, the code feels like last century.
Playwright full end-to-end	Native WebSocket interception, multiple Contexts (separate browser identities), built-in assertion retry mechanisms.

We chose Playwright — not because it’s the latest shiny thing, but because two features hit the sweet spot:

Multi-Context Isolation: Playwright’s BrowserContext internally creates a completely new storage state (cookies/localStorage), naturally isolating two user sessions. It perfectly simulates scenarios like “different identities on the same device” or “same identity across devices”, which is way more reliable than manually clearing cookies.

waitForResponse + custom matcher: LLM responses are streamed as Server-Sent Events (SSE). Playwright can listen to network responses, capture the full answer text, and then use another LLM call for semantic assertion — “does the response contain information provided in the previous round?” — instead of naively waiting for a DOM element.

The architecture has three layers: Data Factory (generates conversation templates with memory anchors) → Execution Layer (Playwright drives the browser through two conversations) → Assertion Layer (structured assertions + LLM-as-Judge semantic checks). If we change the model or memory backend later, we only adjust configurations, not logic.

Core Implementation

First, the dependencies:

pip install playwright pytest-asyncio openai
playwright install chromium

Step 1: Data Factory — Giving Every Test a Verifiable "Memory Anchor"

What this code solves: Each test generates a unique user profile, ensuring that written memory can be precisely traced back and preventing cross-test contamination.

# test_data_factory.py
import uuid
from dataclasses import dataclass, field

@dataclass
class MemoryAnchor:
    """每次测试生成的唯一记忆锚点"""
    user_name: str
    city: str
    budget: int
    session_id: str = field(default_factory=lambda: f"test-{uuid.uuid4().hex[:8]}")

    @classmethod
    def generate(cls) -> "MemoryAnchor":
        """生产随机但可验证的记忆数据"""
        return cls(
            user_name=f"用户{uuid.uuid4().hex[:4]}",  # 确保不重名
            city="深圳",
            budget=5000,
        )

    @property
    def first_message(self) -> str:
        """第一轮对话消息——向 Agent 植入记忆"""
        return f"你好，我是{self.user_name}，我在{self.city}，预算{self.budget}块"

    @property
    def second_message(self) -> str:
        """第二轮对话消息——隐式测试记忆检索"""
        return "我之前说的预算能买什么？需要我重复一遍吗"

    @property
    def expected_memory_items(self) -> list[str]:
        """第二轮回答中应该出现的关键记忆项"""
        return [self.user_name, self.city, str(self.budget)]

Here, we’ve bound “what to say in the first round”, “what to ask in the second round”, and “what the second response should contain” all into one MemoryAnchor object. If the test fails, you can instantly pinpoint it by the session_id in the logs — I added that after a few bitter lessons.

Step 2: Core Test Engine — Two-Round Conversations + WebSocket Interception

What this code solves: Use Playwright’s multi-Context capabilities to simulate two independent sessions, intercept WebSocket messages to capture the LLM’s full streamed response, rather than waiting for DOM rendering.

# test_memory_e2e.py
import pytest
import asyncio
import jso

DEV Community