From 30-Minute Manual Regression to 2-Minute Automation: Uncovering 3 Hidden LLM Memory Bugs

#python #programming

At 2 a.m., a WeCom message popped up: “Bro, the ChatBot forgot the user’s name again. In the fifth turn it called itself ‘Xiao Ming,’ and by the eighth turn it had changed to ‘Mr. Wang.’” I jolted upright in bed and checked the conversation logs—sure enough, the memory broke after a tool call in the seventh turn. This kind of bug required 30 minutes of manual regression testing, typing into the browser while mentally cursing “Didn’t we already fix this?” Even worse, manual verification always missed one or two turns, leaving you uncertain after every test. That was until I set up an automated memory consistency test using Playwright + Pytest: it runs through the core forgetting scenarios in 2 minutes and, as a bonus, caught 3 hidden bugs lurking in the streaming responses. Today, let me share how this “off-work savior” came to life.

Breaking Down the Problem: Why LLM Memory Testing Drives Testers Crazy

Memory in LLM applications isn’t just storing a variable; it involves injecting the conversation history into the prompt so the LLM can answer based on context. Common implementations include LangChain’s ConversationBufferMemory, ConversationSummaryMemory, or custom sliding-window approaches. The essence of consistency testing is to verify after multiple interactions that the LLM still remembers information explicitly established earlier.

Typical pain points:

Regression testing hurts: A full long-conversation scenario (e.g., a user registers, chats for 20 turns with clarifications and tool calls, then asks about their name) takes at least 5–10 minutes manually. Ten scenarios mean an hour.
Assertions rely on human eyes, and missing means wasted: Did the 18th turn still mention “Your birthday is May 20th”? Staring long enough, anyone zones out, especially with streaming text that appears bit by bit.
Regular API tests aren’t enough: Calling the LLM’s completion API directly doesn’t cover issues that only surface in real environments, like frontend rendering, network jitter, or streaming interruptions.

This forces us to adopt end-to-end automation, combining browser operations, real conversation flows, and assertions in one go.

Solution Design: Why Not Selenium/Cypress

Our tech stack is Python backend + React frontend. Ideally, backend developers should be able to write and maintain tests easily. Here’s the comparison:

Selenium: Veteran, but the API isn’t modern, wait mechanisms require manual WebDriverWait, often fails with streaming rendering, and it launches slowly—not ideal for frequent regression.
Cypress: Great, but JavaScript ecosystem. For our full‑stack Python team, it meant maintaining an extra Node environment, steep learning curve, and less flexibility for multi‑page or cross‑origin scenarios than Playwright.
Playwright: Native async/await, intelligent auto‑wait, built‑in trace viewer, and the ability to intercept network requests to mock APIs. Combined with Pytest fixtures and parametrization, different memory scenarios become individual test cases, runnable in parallel with a single click.

The architecture is straightforward: a conftest.py initializes the browser and wraps a utility function that sends a message and waits for the reply to complete; in the test file, each function represents a memory scenario, simulating N turns of conversation with distractions, then using expect().to_contain_text() to assert the LLM’s reply retains the key information. To cope with frequent frontend changes, we required frontend developers to add data-testid attributes to key elements, making selectors robust.

Core Implementation: From Opening the Browser to Asserting Memory

The code is split into two layers: the shared fixture conftest.py and the test cases in test_memory.py.

This code solves: shared browser and page utilities so each test case doesn’t repeat startup logic.

# conftest.py
import pytest
from playwright.sync_api import sync_playwright, Page, expect

@pytest.fixture(scope="session")
def browser():
    """
    session级别，整个测试会话只启动一次浏览器。
    """
    with sync_playwright() as p:
        # headless=False 能看到界面，便于调试；CI上换成 True
        browser = p.chromium.launch(headless=True)
        yield browser
        browser.close()

@pytest.fixture
def page(browser):
    """每个测试用例独立的页面上下文，互不干扰"""
    context = browser.new_context(locale="zh-CN")
    page = context.new_page()
    yield page
    context.close()

@pytest.fixture
def chat(page):
    """
    封装对话操作：发送一条消息，并等待本次回复完整出现在页面上。
    返回本次AI回复的完整文本内容。
    """
    def _send_and_get_reply(message: str, timeout: int = 15000) -> str:
        # 输入框
        input_box = page.locator('[data-testid="chat-input"]')
        input_box.fill(message)
        # 发送按钮
        page.locator('[data-testid="send-btn"]').click()

        # 等待最后一条AI消息出现且不再变化——这里用独特的“消息气泡 + 完成图标”组合来保证流式输出结束
        # 前端约定：流式输出完成时，消息气泡内会多出一个 [data-testid="typing-indicator"]，消失即完成
        last_msg = page.locator(
            '[data-testid="message-bot"] >> nth=-1'
        )
        # auto-wait：先等到元素可见
        expect(last_msg).to_be_visible(timeout=timeout)
        # 等待打字指示器消失，表示流式结束
        typing_indicator = last_msg.locator('[data-testid="typing-indicator"]')
        expect(typing_indicator).to_be_hidden(timeout=timeout)
        # 返回完整的文本内容
        return last_msg.text_content()

    return _send_and_get_reply