DEV Community

BAOFUFAN
BAOFUFAN

Posted on

LLM Memory Consistency Testing: 3 Pitfalls with Playwright + pytest (And 8 Hours of Debugging)

#python #programming

At 1 AM, our test team's chat (Feishu) exploded: "The user just said they're vegetarian, and the AI immediately recommended braised pork—where is the memory stored? This is a critical production bug!" Staring at the flood of messages, I realized end-to-end memory validation was still a manual wasteland: before every release, QA had to simulate conversations by hand. One wrong click meant starting over, production data was off-limits, and the results were never truly trustworthy. That night I decided: I had to build a fully automated, closed-loop test for memory storage consistency using Playwright + pytest. After stepping into three deep pitfalls, the entire regression test went from 20 minutes down to 3 minutes, and no one ever had to stare at a chat window hunting for bugs again.

Why memory storage deceives so easily

Memory in large models typically relies on a backend memory service combined with front-end session caching. A typical scenario:

  • User inputs "My name is Wang Xiaoming" → the backend calls the memory storage API, writing user_name: Wang Xiaoming
  • User asks "What’s my name?" → the chat API retrieves from the memory service and returns "Wang Xiaoming"

Problems fall into three categories:

  1. Write loss: The API was called but the memory didn’t actually persist in the database;
  2. Retrieval error: The data exists but the retrieval picks up the wrong key or gets overwritten;
  3. Front-end session cache interference: localStorage / sessionStorage retains an old memory and doesn't read from the backend.

Manual testing can only sample one or two conversations and can't precisely verify cross-session persistence. Regular API unit tests can't cover front-end cache paths either. What we need is a browser-based end-to-end closed loop: simulate a real conversation to write memory, clear the session, then recreate a new session to query—fully automated and assertable.

Why Playwright + pytest instead of Selenium

When choosing tools, I had three paths:

  • Pure API testing (requests / httpx): Cannot test front-end caching, and many memory APIs require complex authentication. Bypassing the front-end loses session context.
  • Selenium + pytest: It works, but element wait mechanisms are primitive. LLM responses are often streamed and rendered progressively; Selenium's explicit waits can easily be defeated by dynamic DOM.
  • Playwright + pytest: Playwright has native support for auto-waiting, network interception, and multi-context isolation. pytest’s fixture ecosystem can manage browser, page, and context directly, making it a natural fit for the "create new session → write → destroy → new session → query" workflow.

The architecture is simple: each test class represents a memory dimension (user name, preferences, dietary restrictions, etc.), and test methods follow a four-step choreography: write_memory → teardown_session → new_session → verify_memory. pytest fixtures create and destroy browser contexts, ensuring complete isolation. Meanwhile, conversation templates are data-driven; a single JSON file can add or remove test cases.

Core implementation

1. Browser management and session isolation (fixture layer)

This piece solves the problem of "a clean browser context per test to prevent memory cross-contamination."

# conftest.py
import pytest
from playwright.sync_api import sync_playwright

@pytest.fixture(scope="function")   # scope="function" 保证测试间绝对隔离
def browser_context():
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)   # CI 环境必须 headless
        context = browser.new_context(
            storage_state=None,      # 无任何缓存
            locale="zh-CN"
        )
        page = context.new_page()
        yield page, context
        context.close()
        browser.close()
Enter fullscreen mode Exit fullscreen mode

Why it matters: If you use module or session scope, the localStorage and cookies written by the previous test case will bleed into the next one, causing memory assertions to falsely pass—they’re actually using data from the old session. This cost me a full 3 hours to track down.

2. Chat helper utility (encapsulating reusable interaction logic)

This part solves the problem of "how to determine a complete response under streaming output."

# chat_helpers.py
from playwright.sync_api import Page, TimeoutError as PlaywrightTimeout

def send_message(page: Page, text: str):
    """向聊天输入框发送消息并回车"""
    input_el = page.locator("textarea[data-testid='chat-input']")
    input_el.fill(text)
    page.locator("button[data-testid='send-btn']").click()

def wait_for_reply_contains(page: Page, keyword: str, timeout: int = 15000) -> bool:
    """
    大模型回复是流式渲染，不能简单 wait_for_selector，
    这里轮询最新的 assistant 气泡文本，直到包含 keyword 或超时。
    """
    try:
        page.wait_for_function(
            f"""
            () => {{
                const bubbles = document.querySelectorAll('[data-role="assistant"]');
                if (!bubbles.length) return false;
                const latest = bubbles[bubbles.length - 1].innerText;
                return latest.includes("{keyword}");
            }}
            """,
            timeout=timeout
        )
        return True
    except PlaywrightTimeout:
        return False
Enter fullscreen mode Exit fullscreen mode

Pitfall alert: If you use page.wait_for_selector("text=keyword") directly, streaming output can easily lead to partial matches and flaky test failures. Always poll the latest bubble content until the expected keyword appears.

Top comments (0)