From 30% False Positives to Zero: How I Fixed Our LLM Memory Regression Tests

At 2 AM my phone buzzed more violently than any production deploy alert. I opened the notification — the test suite had gone red. Of the 142 failed cases, 103 were flagged as “conversation content mismatch.” I checked the logs: the actual response was “Sure, you mentioned you like The Three-Body Problem last time,” but the assertion expected “Sure, I remember you like The Three-Body Problem.” The only difference was “you mentioned … last time,” and yet this tiny phrasing variation from the LLM was tearing our regression suite to pieces. Right then I knew: if we didn’t change our approach, our CI pipeline would become a “crying wolf” farce and nobody would trust the tests anymore.

Breaking down the problem: why do ordinary tests keep giving false positives?

The setup is straightforward. Our LLM-powered chat product has a “memory” feature — when a user mentions their favorite book or personal preferences, the model can actively recall that information in a later session. After launch, the memory module evolved fast; every release might touch the storage schema, caching strategy, or the recall prompt. We needed an automated regression suite that could guarantee “whatever was remembered stays visible and correct after a page refresh — nothing gets lost or scrambled.”

Our first testing strategy felt natural: use Playwright to simulate a conversation → send a message like “My favorite book is The Three-Body Problem” → refresh the page → send “Do you remember my favorite book?” → check whether the reply contains The Three-Body Problem. That last step is where everything fell apart: we had tied our assertions to the exact wording the LLM generated. Even a minor model fine-tune, prompt adjustment, or temperature change could lead to a different phrasing, breaking the text assertion completely. We tried using Sentence-BERT to compute semantic similarity and set a threshold; that brought the false-positive rate down from 50 % to 30 %. But no matter how much we lowered the threshold, borderline cases kept appearing, and false positives persisted.

The root cause is: you shouldn’t test what the model said — you should test what the system stored and displayed. For our memory feature, the final rendering already lives in the chat history bubbles in the UI. If our regression tests only verify that the text inside those bubbles after a page refresh matches the original message exactly, then we decouple completely from LLM randomness — zero false positives.

Designing the solution: move assertions to the UI history

Since memory is rendered in the chat history, we only need three steps:

1. Send a specific message, e.g. set memory: favorite_book=三体
2. Refresh the page (or log in again) and wait for the history to load
3. Assert that the text inside a particular history bubble exactly equals the original message you sent.

This is pure string equality with no model interference — the false-positive rate is zero. On the technical side:

• Playwright fits modern single‑page apps better than Selenium. It has smart waiting, network monitoring, and handles websockets and SPA routing with ease.
• pytest’s fixture mechanism is perfect for managing browser instances and login state, and parametrization lets us effortlessly cover multiple memory items.
• Why not just call the API directly? Because rendering the history depends on how the frontend parses the storage format (timestamps, metadata deduplication). A correct backend doesn’t guarantee a correct UI; end‑to‑end regression must verify the final DOM.

To achieve truly zero false positives we also established a convention: all test messages use the fixed format set key: value, and assertions are exact equality. No matter how the UI team changes bubble styles, as long as the text node content is correct our tests stay rock solid.

Core implementation: putting the plan into practice step by step

1. User login fixture and data isolation

The conftest.py below solves two problems: every test gets an isolated account so parallel tests don’t pollute each other’s memory, and page.context.storage_state() persists the login state so we don’t waste time logging in repeatedly.

# conftest.py
import pytest
from playwright.sync_api import Page, BrowserContext, Playwright

@pytest.fixture(scope="session")
def browser_context(playwright: Playwright) -> BrowserContext:
    # 启动无头浏览器，关闭缓存以确保刷新后拉取最新历史
    browser = playwright.chromium.launch(headless=True)
    context = browser.new_context(
        viewport={"width": 1280, "height": 720},
        # 禁止缓存，防止历史记录未刷新导致漏测
        ignore_https_errors=True,
        storage_state=None,
    )
    yield context
    context.close()
    browser.close()

@pytest.fixture
def logged_in_page(browser_context: BrowserContext, request: pytest.FixtureRequest) -> Page:
    # 使用参数化动态生成唯一用户，避免数据交叉
    user_id = request.node.name  # 例如 test_basic_memory，确保唯一
    page = browser_context.new_page()
    page.goto("https://your-llm-app.com/login")
    # 简化：假设输入邮箱即可注册/登录
    page.fill("input[data-testid='email-input']", f"{user_id}@test.com")
    page.click("button[data-testid='login-btn']")
    page.wait_for_selector("[data-testid='chat-area']")
    yield page
    # 测试后清理：可选把该用户记忆数据删除（调用 API 或用 UI）
    page.close()

2. Wrapping memory sending and history verification into utility functions

The utility below sends memory messages with a fixed prefix and extracts all text nodes from the history container. This makes the subsequent assertions — whether in or exact match — so concise that the expected result is obvious at a glance.

# utils.py
from playwright.sync_api import Page

MEMORY_PREFIX = "set memory:"

def send_memory(page: Page, key: str, value: str) -> None:
    """发送一条记忆设置消息，格式：set memory: key=value"""
    payload = f"{MEMORY_PREFIX} {key}={value}"
    page.fill("[data-testid='message-input']", payload)
    page.click("[data-testid='send-btn']")
    # 等自己的气泡出现
    page.wait_for_selector(f"[data-testid='m`