Cutting LLM Memory Regression Bugs by 90% with Playwright + pytest

#python #programming

It was 2 AM when I was jolted awake by a barrage of alerts: “User conversation history lost,” “AI doesn’t remember the last thing I said.” I checked the monitoring dashboards—memory storage service was fine, database connections were all healthy. The culprit turned out to be a bizarre async issue on the front end: the memory data was being sent but never actually persisted. We fixed this bug three times. Every time we thought it was resolved, it reappeared in a new form with the next deployment. That cycle only stopped once we built an end-to-end memory stability test suite using Playwright + pytest. The monthly count of these regression bugs dropped from seven or eight to almost zero.

Breaking down the problem: why backend API tests couldn’t catch memory storage failures

Our product is an LLM-based chat application with memory: after a user logs in, the AI remembers context across multiple conversation turns. The memory lives on the server, but the write trigger lives on the front end. When a user sends a message or the AI finishes a reply, the front end calls an instrumentation endpoint to store a summary of the current conversation. The pain point was exactly this “front-end trigger” step — the HTTP request might be fired, but due to page unloading, async race conditions, or framework state glitches, the data never persisted. Backend API tests only verified that the endpoint returned 200; they covered none of these real-world scenarios.

What makes it even trickier is that verifying whether memory was stored successfully requires checking whether the user can see the previous context when they reopen the conversation. This is inherently a cross-session, cross-time, end-to-end flow. You can’t catch it by hitting an API with Postman or pytest alone. You have to simulate actual browser behavior and go through the entire sequence: “send a message → close the page → reopen → check history.”

Conventional backend integration tests break down here. We tried writing scripts with Selenium, but as the number of test cases grew, execution became painfully slow, and the environment setup was so fragile that colleagues gave up maintaining it. The result: before every release, someone had to manually click through the critical paths in a browser — time-consuming, exhausting, and easy to miss things.

Solution design: why Playwright + pytest, and nothing else

We needed an end-to-end testing solution that was fast, stable, and easy to maintain — one that could realistically simulate user interactions in the browser and verify the memory content displayed on the UI. The main candidates were:

Selenium + unittest: the veteran combo, but slow execution, reliance on explicit sleeps for waiting, and painful environment setup (especially chromedriver version matching). Ruled out.
Cypress: well-loved in the front-end community, but our backend is Python-based. Test cases require heavy data preparation and API assertions, which are more natural to write in Python and reduce the team’s learning curve. Additionally, Cypress has limitations with multi-tab and cross-origin scenarios.
Playwright + pytest: from Microsoft, fast startup, built-in smart waiting, native support for multiple pages and contexts, and the ability to record interactions and generate code. Most importantly, the pytest-playwright plugin integrates Playwright into your existing test framework with a single command, automatically capturing screenshots and traces on failure — troubleshooting becomes ten times faster than scrolling through logs.

Once we made our choice, the architecture looked like this:

pytest organizes test cases: grouped by memory storage scenarios (in-session memory, cross-session persistence, multi-user isolation, etc.), using fixtures to manage browser contexts and data preparation.
Playwright simulates the user: browser.new_context() creates an isolated session (equivalent to a fresh browser profile). context.storage_state() persists login state so it can be reused across cases.
Custom assertions: instead of just checking API responses, we directly parse the DOM to confirm that the expected historical messages appear on the chat interface.
CI integration: a headless browser runs in GitLab CI; the full suite executes automatically on every merge request, with screenshots and traces uploaded to artifacts on failure.

There’s a nuance that official documentation doesn’t emphasize: browser_type.launch() starts in incognito mode by default. To test cross-session persistence, you have to use launch_persistent_context() with a user data directory so you can truly simulate “close the browser and reopen it, with local storage and cookies intact.” We’ll dig into that later.

Core implementation: building a reliable memory regression test from scratch

1. Environment setup: getting the tests to run

Install dependencies:

pip install pytest-playwright playwright
playwright install chromium

Create a conftest.py in the project root and define basic fixtures. The code below ensures that each test case gets an already-logged-in browser context, and cases are automatically isolated so they don’t pollute each other.

# conftest.py
import pytest
from playwright.sync_api import sync_playwright

@pytest.fixture(scope="session")
def browser():
    # Launch the browser process once for the whole test session to save time
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)  # Use headless in CI
        yield browser
        browser.close()

@pytest.fixture
def logged_in_context(browser):
    """
    Create an independent browser context for each test case,
    simulate a fresh logged-in user environment,
    and save the login state for reuse by subsequent cases.
    """
    context = browser.new_context(
        viewport={"width": 1280, "height": 720},
        locale="zh-CN"
    )
    page = context.new_page()

    # ---- Login flow ----
    page.goto("https://your-app.com/login")
    page.fill('input[name="email"]', "testuser@example.com")
    page.fill('input[name="password"]', "testpassword")
    page.click('button[type="submit"]')
    # Wait for a characteristic element after login instead of a fixed sleep
    page.wait_for_selector(".chat-sidebar", timeout=10000)

    # Save login state; later cases can use it to create already-logged-in contexts
    context.storage_state(path="auth.json")

    yield context
    context.close()