I Spent 8 Hours Debugging an LLM Context Amnesia Bug — Here’s How I Automated Regression Testing

In the middle of the night, I got bombarded with pings in our project group. A user reported that they had spent half an hour yesterday discussing travel plans with the bot, but when they continued the conversation this morning the bot asked, “Where would you like to go?” — right back to square one. I looked at the code: the new version had rewritten the context truncation logic to control token usage, and historical messages were being mindlessly thrown away. Manually replaying a dozen memory scenarios would take half an hour and might not even reproduce the issue. By 3 a.m. I made the call: I had to build a fully automated context regression test with Playwright.

Breaking Down the Problem

What “remembering context” really means in an LLM application is that the front end sends the multi-turn conversation history (either verbatim or after summarisation) along with the next request. At least four things can go wrong along this path:

1. Front‑end state management: the message array gets cleared on page refresh or route changes.
2. Truncation strategy: the back end or a middleware chops messages strictly by token count, slicing away the earliest critical information (name, preferences) the user shared.
3. Async timing: the user sends another message before the AI has finished replying, which scrambles the concatenation order of the history.
4. Prompt assembly: in multi‑agent setups the format of the system prompt mixed with the chat history is wrong, causing the entire memory to collapse.

The typical manual testing approach is to chat for a dozen turns and see whether the AI remembers. But LLM responses are non‑deterministic, so after a few rounds you just want to give up. Every release could introduce a new regression, and manual testing simply cannot keep up. We needed automatic regression that could simulate a continuous conversation and assert “did it remember or not?”

Solution Design

Why Playwright?

• Must go through the UI: context‑memory bugs often hide inside the payload the front end sends to the back end. Pure API testing cannot catch them — you have to simulate a real browser sending messages.
• WebSocket / streaming responses: Playwright has built‑in network waiting and smart auto‑waiting. When the AI is outputting token by token, you can wait until the element’s content stabilises before making an assertion. Selenium is much weaker at this.
• Context isolation: each test runs in its own browser context, independent and parallel‑ready.

Why not other options?

• Cypress: also strong, but our back‑end toolchain is heavy on Python. Playwright’s Python API integrates seamlessly with pytest, so maintenance costs are low.
• API recording and replay: fine for the first call, but the request body grows differently with each turn in a multi‑turn conversation. Replaying recorded payloads would duplicate history, creating an artificial context flow rather than the real one.
• Directly mocking the LLM: possible, but the regression test aims to verify “what exactly did the front end send?” Mocking the LLM would hide changes in the API contract.

Overall plan: use Playwright to open the chat page → send messages according to a predefined dialogue sequence → wait for the assistant’s reply after each turn → assert that the memory was correctly carried over by checking for keywords or specific markers.

Core Implementation

This snippet handles browser startup and the first memory verification

We define the basic test structure, launch the browser, open the page, send the first message (telling the AI the user’s name), and then assert that the assistant’s reply contains that name — proving that the first round of memory is established.

from playwright.sync_api import sync_playwright
import pytest

CHAT_URL = "http://localhost:3000/chat"  # 被测页面

def send_message(page, text: str):
    """向聊天框发送一条消息并回车"""
    page.fill("[data-testid='user-input']", text)
    page.click("[data-testid='send-btn']")

def wait_for_assistant_reply(page, timeout=15000):
    """等待助手回复区域出现新内容"""
    # 定位到最新的助手消息元素（假设每条消息有独立容器）
    locator = page.locator("[data-testid='assistant-message']").last
    # Playwright 自动等待元素可见且有文本内容
    locator.wait_for(state="visible", timeout=timeout)
    # 额外等待流式输出结束：内容长度稳定200ms不再变化
    page.wait_for_function(
        "selector => {"
        "  const el = document.querySelector(selector);"
        "  return el && el.textContent.trim().length > 0;"
        "}",
        "[data-testid='assistant-message']",
        timeout=timeout
    )
    return locator.text_content()

def test_memory_single_turn():
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)  # CI环境必须headless
        page = browser.new_page()
        page.goto(CHAT_URL)

        # 第一轮对话：告诉AI名字
        send_message(page, "我叫张三，帮我记录一下")
        reply = wait_for_assistant_reply(page)
        assert "张三" in reply, f"首轮应记住名字，实际回复: {reply}"

        browser.close()

This snippet automates multi‑turn memory‑chain verification

We put the dialogue sequence into a loop, sending a new question each turn and asserting that the reply contains information mentioned earlier. This covers cross‑turn context retention.

def run_memory_chain(page, dialogs: list[tuple[str, str]]):
    """
    dialogs: [(发送内容, 期望回复中包含的关键词), ...]
    每轮自动发送并检查记忆锚点。
    """
    results = []
    for i, (user_msg, expected_keyword) in enumerate(dialogs):
        send_message(page, user_msg)
        reply = wait_for_assistant_reply(page)
        # 断言核心：关键词必须在回复中，否则说明上下文丢失
        assert expected_keyword in reply, (
            f"第{i+1}轮记忆失败: 期望包含'{expected_keyword}'，"
            f"实际回复: {reply[:200]}"
        )
        results.appen