From Manual to Playwright+Pytest: Boosting AI Memory Test Coverage from 20% to 96%, Slashing Regression Time by 90%

At 3 a.m., my phone screamed me awake. The founder had dropped a screenshot in the group chat: a user asked, "Where did we leave off yesterday?" and the AI, as if drinking from the river Lethe, replied, "Please describe your issue again."

Production incident. The memory store was gone.

After half a day of digging, the dev team pinpointed a chilly edge-case race condition: message ordering chaos caused a memory write to fail. Manual testing never covered the scenario of multiple sessions interleaving writes. That night, while patching the bug, I made a resolution: the E2E tests for chat memory must be automated and must cover every twisted scenario.

Breaking down the problem: why is testing AI memory storage so hard?

AI conversation memory isn't just "save a field." The core flow is: User A speaks → generate a reply while extracting memory → persist to storage (Redis/Postgres) → recall and build context in the next turn or a new session. Testing needs to simulate:

• Correct extraction and updating of memory across multiple turns;
• Memory persistence across sessions (close the browser, reopen, still remembered);
• Memory overwrites and misplacements under concurrent requests;
• The timing of asynchronous writes in the front-end/back-end interaction.

Manual testing typically means a person clicking around: type a few lines in the chat, close the tab, reopen it, see if it remembers. It’s easy to miss critical paths, and it takes an appalling amount of time—a full regression cycle once took over 4 hours, and no one dared guarantee complete coverage. We desperately needed an automation solution that could simulate real browser interaction, provide precise assertions, and run in CI.

Solution design: Playwright + pytest, and why nothing else

We ultimately chose Playwright as the browser driver, paired with pytest for test management and assertions, outputting Allure reports, all running on GitHub Actions.

Why not Selenium? It’s slow. Selenium requires separate WebDriver installation, its waiting mechanisms rely entirely on implicit/explicit wrappers, and it often fails falsely due to network jitter. Playwright has built-in auto-waiting, network interception, and a trace viewer—the debugging experience is leagues ahead. What about Cypress? Despite a great developer experience, it doesn’t support multiple tabs or parallel browsers, and cross-session tests must open a new tab or browser context, so it was immediately ruled out. pytest, meanwhile, fits our Python stack better than Mocha/Jest, with native parameterization and fixture state management.

Architecturally, we built three layers of abstraction:

1. Page Object – encapsulates chat page elements and actions (send_message, get_last_reply...)
2. browser fixture – manages the browser context, isolating storage state (storageState) per test
3. memory validator – a reusable assertion utility that polls for memory recall to take effect, handling async writes.

Core implementation: writing memory storage tests that actually run

First, solve global state and authentication – use conftest.py to provide a persistent authenticated context, avoiding repeated login per test.

# conftest.py
import pytest
from playwright.sync_api import sync_playwright, BrowserContext

@pytest.fixture(scope="session")
def browser_context() -> BrowserContext:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)  # headless in CI
        # load saved login state from file, avoid repeated login, save ~8 sec per test
        storage_state = "auth.json"
        context = browser.new_context(storage_state=storage_state)
        yield context
        context.close()
        browser.close()

The first critical case: cross-session memory recall

This code simulates a user saying "My name is Xiao Ming" in one session, then asking "What's my name?" in another session, and verifies the AI truly remembered the name.

# test_memory_cross_session.py
from playwright.sync_api import Page, expect

def test_memory_remembered_across_sessions(browser_context):
    # Session 1: establish memory
    page1 = browser_context.new_page()
    page1.goto("https://chat.example.com")
    page1.fill('[data-testid="chat-input"]', "我叫小明，喜欢喝咖啡")
    page1.click('[data-testid="send-btn"]')
    # wait for reply to appear, proving the message was processed
    expect(page1.locator('[data-testid="msg-bubble"]').last).to_contain_text("喜欢喝咖啡", timeout=10000)

    # Session 2: verify memory recall in a new tab
    page2 = browser_context.new_page()
    page2.goto("https://chat.example.com")
    page2.fill('[data-testid="chat-input"]', "你还记得我叫什么吗？")
    page2.click('[data-testid="send-btn"]')

    # critical: async memory write may have a delay, cannot assert directly; need smart waiting
    expect(page2.locator('[data-testid="msg-bubble"]').last).to_contain_text("小明", timeout=15000)

The second case: multi-turn memory updates without loss

This is the scenario that originally caused the outage: sending messages rapidly in succession—will memory extraction get jumbled?

def test_rapid_conversation_memory_update(page: Page):
    page.goto("https://chat.example.com")
    # simulate the user quickly sending three messages
    messages = [
        "我今年30岁",
        "我住在杭州",
        "我的猫叫豆包"
    ]
    for msg in messages:
        page.fill('[data-testid="chat-input"]', msg)
        page.click('[data-testid="send-btn"]')
        # wait for server receipt of each message to avoid queue pile-up
        expect(pa