Replacing Manual LLM Memory Checks with Playwright + pytest Boosted Recall Accuracy Verification by 10x

#python #programming

It was 2 AM when the product manager, frantically @-ing me in the group chat, wrote: “A user told the AI assistant earlier that their birthday is March 15th, but when they asked again just now, the AI said it didn’t remember. Did the conversation memory get lost?” Rubbing my eyes, I opened the log system and started digging through tens of thousands of conversation records—like looking for a needle in a haystack. Right then I knew: our manual spot-checking approach had to go.

Breaking down the problem

Once an LLM has “memory,” storing and recalling multi-turn conversations becomes a core part of the user experience. Something a user says today must be remembered when they ask again three days later. But if the storage doesn’t persist, the vector search misses, or the context trimming strategy is wrong, you end up with a “phantom recall.” Our system flow was: use embeddings to write conversation summaries into a vector database, retrieve top_k results when needed, and stitch them into the prompt. How did we test this conventionally? Open the frontend, manually type a few lines like “My name is Zhang San, I’m in Beijing,” then wait, ask again later “What’s my name? Which city am I in?”, inspect the reply with our eyes, and finally check the database and logs to confirm whether that memory was actually retrieved. One scenario took at least five minutes; testing 20 edge cases would eat up well over an hour, and your eyes would inevitably get tired and miss things. Sure, unit tests covered vector retrieval and insertion, but they didn’t cover the full browser-based chain—from the frontend sending a message, to the backend assembling the prompt, to the LLM’s response being rendered on the page. If any link in that chain breaks, all the user sees is “I forgot.” We needed an end-to-end, repeatable, and assertable automation solution that directly mimics real user browser interactions, handing the verification of conversational memory storage and recall accuracy over to code.

Solution design

For browser automation, the three main choices are Selenium, Cypress, and Playwright. Selenium is too old and slow—writing async waits with it drives you up the wall. Cypress has a nice ecosystem but limited support for multiple tabs and cross-origin scenarios, and some of our memory tests require switching tabs to verify different conversations. In the end I chose Playwright + pytest. Playwright comes with automatic waiting, network interception, trace viewer, and even built-in video recording—debugging issues with replays is a joy. pytest is the most flexible Python testing framework: fixtures, parameterization, and a rich plugin ecosystem all make life easier.

The architecture is straightforward: each test case is an independent multi-turn conversation scenario. A pytest fixture creates a brand-new browser context (isolated cookies/sessions). Playwright opens the chat page, simulates a user sending several rounds of messages, waits for the AI’s reply to appear in the DOM, and asserts that the reply text contains the expected memory keywords. When needed, you can also intercept API requests using page.route to directly inspect the backend’s returned prompt or vector search results, enabling white-box validation. Why not just test the HTTP endpoints directly? Because what real users see is the rendered interface. Some bugs occur exactly when the backend returns the correct content but the frontend truncates, garbles, or covers it with an interactive component—defects that API testing would never catch.

Core implementation

Here’s the code with all imports included, ready to drop into your project. The application under test is a simple chat interface with an input field id="user-input", a send button .send-btn, and the AI’s replies appearing inside elements with the class .ai-message.

First snippet: a pytest fixture that gives each test case a clean browser context

This solves test isolation—tests must not share cookies or sessions, otherwise memories leak across cases.

# conftest.py
import pytest
from playwright.sync_api import sync_playwright

@pytest.fixture(scope="function")
def browser_context():
    with sync_playwright() as p:
        # Start a headed browser for debugging; switch to headless=True in CI
        browser = p.chromium.launch(headless=False, slow_mo=100)
        # Each test gets its own context — no cookie/localStorage interference
        context = browser.new_context()
        page = context.new_page()
        page.goto("http://localhost:3000/chat")
        # Wait for the page to fully load and the chat input to be visible
        page.wait_for_selector("#user-input", state="visible")
        yield page
        # Clean up after the test to avoid browser process leaks
        context.close()
        browser.close()

Second snippet: a basic memory recall test — two rounds of conversation, asserting that the memory is correctly recalled

This test simulates a user first telling the AI some personal information, then switching topics and asking about it later, verifying that the AI remembered. The key is using page.fill to type text, page.click to send, waiting for the AI reply node to appear in the DOM, and finally using assert to check the text.

# test_memory_basic.py
import pytest
from playwright.sync_api import expect, Page

def test_recall_basic_info(browser_context: Page):
    page = browser_context

    # First round: inject memory information
    page.fill("#user-input", "我叫王小明，我是一名后端工程师，公司在北京。")
    page.click(".send-btn")
    # Wait for the AI reply node to appear, up to 15 seconds (the model might be slow)
    page.wait_for_selector(".ai-message", state="visible", timeout=15000)
    # Grab the first reply text — usually an acknowledgment, which we consume here
    first_reply = page.text_content(".ai-message:last-of-type")

    # Second round: a casual chat line to ensure a context switch
    page.fill("#user-input", "今天天气不错，适合去哪玩？")
    page.click(".send-btn")
    page.wait_for_selector(".ai-message", state="visible", timeout=15000)

    # Third round: critical recall — directly ask about the memorized information
    page.fill("#user-input", "你还记得我叫什么名字，在哪里工作吗？")
    page.click(".send-btn")
    page.wait_for_selector(".ai-message", s