DEV Community

BAOFUFAN
BAOFUFAN

Posted on

I Wrote a Playwright Script to Test LLM Long-Term Memory — and Found 3 Critical Bugs

#python #programming

It was 1 a.m. when the PM dropped a screenshot into the group chat: “Your AI assistant forgot the customer’s name again. Third time.” I zoomed in and dragged my finger across the words “Dear user, hello!”. My heart sank — the “long-term memory module” we had just shipped was completely unreliable in the real world. I had manually tested dozens of conversations. It always felt fine, but the moment it hit real users, everything fell apart. I closed Slack, opened my IDE, and decided to build an automated test with Playwright that would slap the system in the face, repeatedly. In the end, the test didn’t just pin down the problem — it also unearthed three fatal framework-level bugs that were making our memory go schizophrenic.

Deconstructing the problem: why manual testing fails for LLM long-term memory

We were building an LLM-powered customer support assistant, with LangChain’s ConversationSummaryBufferMemory handling long-term storage on the backend. The dream was simple: the user’s name, requests, and preferences should be remembered across sessions. The reality? Memory is probabilistic — the model forgets sometimes, the framework messes up sometimes, and edge cases (like concurrent sessions) can cause memory to leak from one user to another. Manual testing feels impossible because:

  • Repetition burns you out: Every regression means holding dozens of conversations again. No human can keep up.
  • No precise assertions: Scanning a response and thinking “hmm, seems like it remembers” is way too subjective. Get one order number wrong and it explodes.
  • Cross-session state is a nightmare to simulate: Cookie, localStorage, and session ID isolation — you just can’t reliably reconstruct that by hand.

Regular unit tests can verify the memory component in isolation, but the true end-to-end memory flow — from user input to memory summarization to cross-session retrieval — only reveals itself through real browser interaction. That’s when I turned to Playwright: it can click pages, log in, operate multiple windows just like a real user, and reliably extract the LLM’s reply text from the DOM for assertions.

Architecture decisions: why not Selenium or call the API directly?

A lot of people asked me: “Why do you need a browser to verify memory? Can’t you just call the /chat endpoint?” If we were only checking model outputs, sure. But the full long-term memory chain involves frontend session management and browser-side state retention. For example:

  • The session ID might be generated by the frontend and stored in sessionStorage. The backend only recognizes that ID.
  • Some memory cleanup logic is tightly coupled with the creation of a new chat session, so you have to simulate a “click new session” action.

So browser testing is unavoidable. Next came tooling:

  • Selenium is too heavy, its async wait mechanism feels dated, and it forces you to write piles of explicit waits for modern SPAs — every time I used it I wanted to smash my keyboard.
  • Puppeteer is great, but my team works primarily in Python. Maintaining a cross-language test suite is a net loss.
  • Playwright for Python offers native async/await (with a synchronous API too), auto-waiting for elements, network interception, and the ability to open multiple browser contexts in a single script to simulate different users. Perfect.

For architecture, we used pytest to drive Playwright. Each test case simulates a full memory flow:

  1. Create a user (log in, initialize a session).
  2. Inject a memory in session A (e.g., “My name is Wang Xiaoming, employee ID 9812”).
  3. Open an entirely new session B (simulating a visit the next day).
  4. Ask a memory-related question, extract the LLM response, and assert that key information is present.

This approach keeps every test case independent and parallelizable.

Core implementation: building a reusable memory verification script step by step

1. Shared fixtures: browser instance and page

This code solves the “start/stop browser for every test” problem. We use a session-scoped fixture to launch Chromium only once.

# conftest.py
import pytest
from playwright.sync_api import sync_playwright

@pytest.fixture(scope="session")
def browser():
    with sync_playwright() as p:
        # headless=False 可以在调试时看到浏览器操作，CI 里设 True
        browser = p.chromium.launch(headless=False, slow_mo=100)
        yield browser
        browser.close()

@pytest.fixture
def page(browser):
    context = browser.new_context()      # 隔离的浏览器上下文，相当于一个全新用户
    page = context.new_page()
    yield page
    context.close()
Enter fullscreen mode Exit fullscreen mode

2. The simplest memory test: store in one session, retrieve in another

This test validates the most basic long-term memory correctness — that something you put in can be pulled out on a different day.


python
# test_memory.py
import re
import pytest

def test_basic_long_term_memory(page):
    page.goto("http://localhost:3000/login")

    # 登录（假设一个简单的表单）
    page.fill("input[name='username']", "alice")
    page.fill("input[name='password']", "secret")
    page.click("button:has-text('登录')")
    page.wait_for_selector(".chat-container")   # 等待主聊天界面出现

    # --- 会话 A：注入记忆 ---
    page.fill("textarea.chat-input", "我叫王小明，工号 9812，负责华东区售后。")
    page.click("button:has-text('发送')")
    # 等待 LLM 回复出现在最后一条消息中（必须确定有文字，不能只是 DOM 挂载）
    page.wait_for_selector(".message.assistant:last-child :text-is('好的')", timeout=10000)

    # 新建会话，模拟跨天
    page.click("button:has-text('新会话')")
    page.wait_for_selector(".chat-container")

    # --- 会话 B：验证记忆 ---
    page.fill("textarea.chat-input", "请问我的工号是多少？")
    page.click("button:has-text('发送')")
    # 关键点：不用 sleep，用文本断言等待
    page.wait_for_selector(".message.assistant:last-child", timeout=10000)
    re
Enter fullscreen mode Exit fullscreen mode

Top comments (0)