Automating LLM Agent Memory Testing with Playwright: 10x Efficiency Boost

A while ago, our team took on a task: regression testing for an enterprise AI assistant, with a focus on its “memory” and context retention. In round 1 you tell it “My name is Zhang San and I work at ByteDance.” By round 5, when you ask “What’s my name? Which company do I work for?”, it must answer correctly. At first, three of us were doing this manually. We’d run at most 30 test cases a day, frequently miss subtle regressions when our attention drifted, and constantly worry about edge cases. Frustrated, I automated the entire workflow with Playwright — now a single machine easily runs 300+ cases a day, and I no longer dread being woken up in the middle of the night to “put out fires”. This post is a recap of that overhaul, and all the code is ready to run.

Breaking down the problem

Our test target is an in‑house LLM Agent web app. Users access it through a browser, can have multi‑turn conversations, and the agent is expected to have “long‑term memory” — remembering user preferences, facts, and recalling them even after the conversation shifts to different topics.

A typical manual test flow goes like this: open the page → type “My name is Zhang San and I work at ByteDance” → chat about unrelated stuff → suddenly ask “What’s my name?” → manually judge if the answer is correct. A few things drive you crazy:

• Long context windows — a single session often spans a dozen turns. By the later rounds, your brain is fried.
• Blurred forgetting boundaries — LLM “forgetting” is rarely a hard drop; it might misremember a single character (“Zhang Shan”), which is painfully easy to miss with the naked eye.
• Combinatorial explosion of scenarios — name, company, preferences, order history… the permutations quickly become impossible to cover manually.
• Interleaved tool calls — the agent supports in‑conversation interruptions like “Book a flight for me”. Crafting that test data by hand is a huge pain.

Classic API‑level testing (calling /chat directly) has two fatal flaws: it skips the front‑end message‑history stitching logic (our app has its own message merging strategy), so you never test the full chain; and the API often returns streaming chunks, making assertions tricky, and you can’t reproduce the final rendered text the user actually sees. End‑to‑end browser automation is the only reliable way.

Design decisions

When picking a tool, I ruled out Selenium immediately. Not because it’s bad — it’s just that our Agent’s front‑end relies heavily on WebSockets pushing streaming tokens. Waiting for dynamic content with Selenium feels clunky; you end up writing piles of WebDriverWait. Playwright natively supports waiting for network idle, text changes in elements, and can even intercept WebSocket frames — plus its Python async API is butter smooth.

The architecture boils down to three simple rules:

1. Tests as configuration – each memory test becomes a “dialogue sequence + final assertion”, managed in YAML, completely decoupled from code.
2. One browser context per session – each test case uses an isolated browser context, keeping localStorage/sessionStorage separate, avoiding cross‑contamination and allowing parallel execution.
3. Smart waiting – no hardcoded sleep(2). Instead, we use Playwright’s expect(page.locator(...)).to_contain_text(...) to wait for the AI’s complete response. More stable than any fixed delay.

Why not use an existing AI testing framework (like DeepEval’s E2E part)? Most of them are still API‑oriented, offer weak support for front‑end rendering and multi‑step interactions, and add extra dependencies that the team has to maintain. Wrapping Playwright ourselves took about 200 lines of code, and we have far more control.

Core implementation

Let’s get to the good stuff. The three code blocks below make up a minimal, runnable memory test script.

First block: page interaction helper. It solves “how to reliably send a message and get back the AI’s complete response”.

import asyncio
from playwright.async_api import async_playwright, expect

async def send_and_get_reply(page, message: str, timeout: int = 15000) -> str:
    """
    向聊天框发送 message，等待 AI 返回完整响应文本
    特别处理流式输出：等待"停止生成"按钮消失，表明回复结束
    """
    # 定位输入框并填入文本，通常 chatbot 都有一个 textarea
    input_box = page.locator('textarea[placeholder*="输入消息"]')
    await input_box.fill(message)

    # 点击发送按钮
    send_btn = page.locator('button:has-text("发送")')
    await send_btn.click()

    # 关键：等待流式输出完成。我们前端在生成时会显示一个"停止"按钮，
    # 生成结束后按钮消失。也可以用其他页面标志，比如光标出现。
    stop_btn = page.locator('button:has-text("停止生成")')
    await expect(stop_btn).to_be_hidden(timeout=timeout)  # 等15秒

    # 获取最后一条 AI 消息的完整文本
    # 假设消息列表最后一条 class="assistant-message"
    last_message = page.locator('.assistant-message').last
    await expect(last_message).to_be_visible()
    return await last_message.inner_text()

Second block: memory test case runner. It solves “how to execute a multi‑turn conversation and run assertions”.

async def run_memory_test(browser, case: dict):
    """
    case 格式:
    {
      "name": "基础记忆-姓名公司",
      "conversations": [
        {"role": "user", "content": "我叫张三，我在字节跳动工作"},
        {"role": "assistant", "ignore": True},  # 不验证
        {"role": "user", "content": "今天天气真好"},
        {"role": "assistant", "ignore": True},
        {"role": "user", "content": "我叫什么？我在哪工作？"},
      ],
      "expected": ["张三", "字节跳动"]  # 最后一条回复必须同时包含这些词
    }
    """
    context = await browser.new_context()
    page = 

Third block: parallel execution entry point. Kick off all the test cases at once. (I’ll leave that as an exercise — it’s really just an asyncio.gather loop over your YAML‑defined cases using the two helpers above.)

With these pieces, we turned a tedious, error‑prone process into something reliable and massively scalable. The team now adds new memory scenarios by simply appending YAML, and the entire suite runs multiple times a day in CI. If you’re testing LLM‑powered chat interfaces, give Playwright end‑to‑end automation a try — your sleep schedule will thank you.