How We Cut Memory Hallucination Debugging from 6 Hours to 30 Minutes with Playwright + pytest

At 1 AM, the product manager dropped a screenshot into the group chat. A user had asked, "How’s that project I mentioned going?" and the AI answered, "You haven’t told me about any project." The user fired back with a chat history screenshot that clearly showed a project description from three days earlier. The PM said: "You call this memory? This is amnesia."

I knew instantly — another hallucination regression. The memory storage module had been touched, but no one had verified it. And running a full regression manually across the entire conversation flow would take at least half a day. That night, after fixing the bug, I committed a Playwright + pytest end-to-end verification script and swore: if anyone ever made me test memory manually again, I’d slam the test report on their desk.

Why memory storage regression is so painful

Our AI application’s killer feature is cross-session memory — everything a user mentions in any conversation gets stored in a persistent memory store (pgvector + vector index) and is automatically recalled in future chats. Sounds great, but the "cross-session" part is exactly where it hurts. To verify that memory really works, you must simulate full multi-turn, multi-session interactions and check that the AI’s responses contain the expected memory fragments.

Our old regression workflow looked like this:

1. Manually call the /chat endpoint via Swagger and paste a message containing personal info (e.g., "I’m working on a project called Phoenix").
2. Wait for the backend to asynchronously chunk the memory, generate embeddings, and write to pgvector.
3. Open a new session and ask, "What was the project I mentioned earlier?"
4. Visually inspect the response for the word "Phoenix."

Every time we touched the memory module — even just adjusting the chunk size — we had to repeat that cycle N times, covering edge cases (Chinese, English, special characters, very long texts, etc.). A complete regression took at least 6 hours, and human eyes easily missed hallucinations (the AI could sound right but substitute the wrong entity). Worse, memory writes are asynchronous, so waiting relied on sleeps; during manual testing we constantly ran into timeouts or incomplete writes, leading to a sky‑high false‑positive rate.

Why Playwright + pytest, and not something else

We needed an end‑to‑end framework that could realistically simulate user interactions in the browser, wait for async results, and assert on natural language fragments. A few options were on the table:

• Pure API tests (requests + pytest): You can hit endpoints directly, but you can’t carry the frontend‑side context (headers, cookies, session IDs in localStorage). Our app streams responses over SSE, which is painful to simulate with plain requests.
• Cypress: Great browser automation, but our backend stack is Python, the team isn’t comfortable with JS/TS, and Cypress’s parallelism and fixture management are less flexible than pytest.
• Playwright + pytest: Playwright ships an official pytest-playwright plugin. You write tests in Python, reuse all of pytest’s fixtures, parametrization, conftest, etc. You control Chromium, grab page DOM or network responses, and can even call backend APIs from inside tests for data setup/teardown. Most importantly, page.wait_for_response lets you wait for async requests to finish naturally, eliminating hard‑coded sleeps.

Architecturally, we split things into three layers:

1. Page interaction layer: Playwright opens the frontend, types into the chat input, clicks send, and reads the AI reply from the streaming output.
2. Verification layer: Extract the AI’s reply from the DOM, assert it contains a known memory word (like "Phoenix"), and use negative assertions to ensure no hallucination (e.g., "Project X" should not appear).
3. Data helper layer: Before a test, seeds known memories directly via the API or cleans the database to guarantee a clean environment.

Core implementation: setting up memory E2E verification from scratch

1. Defining test fixtures to isolate environments

What this solves: each test case needs its own browser context, a logged‑in state, and a cleanup step that wipes memories afterward so tests don’t pollute each other.

# conftest.py
import pytest
from playwright.sync_api import Page, BrowserContext

@pytest.fixture(scope="session")
def browser_context(browser: BrowserContext):
    # 设置全局上下文，如视口大小、忽略HTTPS错误
    context = browser.new_context(
        viewport={"width": 1440, "height": 900},
        ignore_https_errors=True
    )
    yield context
    context.close()

@pytest.fixture(scope="function")
def page(browser_context: BrowserContext) -> Page:
    page = browser_context.new_page()
    # 先登录，获取token存入localStorage
    page.goto("https://ai-app.example.com/login")
    page.fill('input[name="email"]', "[e-mail]")
    page.fill('input[name="password"]', "testpass")
    page.click('button[type="submit"]')
    # 等待登录完成，跳转到聊天页
    page.wait_for_url("**/chat")
    yield page
    # 清理：调用后端API删除该测试用户的所有记忆，防止数据残留
    page.request.delete("https://api.ai-app.example.com/memories/[e-mail]")
    page.close()

2. Wrapping conversation and memory verification into a reusable helper

What this solves: abstracting “send a message → get the reply → assert memory” into a reusable function that supports streaming waits and memory‑fragment checks.

# test_helpers.py
from playwright.sync_api import Page
import re

def send_and_get_reply(page: Page, message: str, timeout: int = 15000) -> str:
    """发送消息并等待AI完整回复（SSE流式结束后提取文本）"""
    page.fill('textarea[data-testid="chat-input"]', message)
    page.click('button[data-testid="send"]')
    # 关键点：等待消息列表中最后一条AI消息的出现，且内部文本不再变化
    # 因为SSE流式更新，我们等待消息气泡出现且包含内容长度稳定
    last_msg = page.locator('[data-testid="message-bubble"]:last-child')
    last_msg.wait_for(state="visible", timeout=timeout)
    # 等待文本长度稳定（即流式结束，简单实现：连续两次长度一致）
    previous_len = -1
    for _ in range(10):
        current_len = len(last_msg.inner_text())
        if current_len == previous_len and current_len > 0:
            break
        previous_len = current_len
        page.wait_for_timeout(500)
    return last_msg.inner_text()

def assert_memory_recall(page: Page, memory_word: str, question: str):
    """发送问题并断言回复中包含记忆词"""
    reply = send_and_get_reply(page, question)
    assert memory_word.lower() in reply.lower(), \
        f"Expected memory word '{memory_word}' not found in reply: {reply[:200]}"

def assert_no_hallucination(page: Page, prohibited_word: str, question: str):
    """断言回复中不包含特定幻觉词"""
    reply = send_and_get_reply(page, question)
    assert prohibited_word.lower() not in reply.lower(), \
        f"Hallucination detected: '{prohibited_word}' found in reply: {reply[:200]}"