From 3 Hours to 3 Minutes: How We 60x’d AI Chat Memory Regression Testing

#python #programming

Monday morning, I barely opened my laptop when the PM dropped a message: “A user reported that yesterday they told the AI their name was ‘John’, but today when asked ‘What’s my name?’, the AI had no clue. Check if the memory module is down, ASAP.” I sighed, opened the browser, and started the manual reproduction ritual—clear cache, send a self-introduction, send a follow-up question, take screenshots, compare… At least 5 minutes per scenario. The PM handed me 8 scenarios, plus switching between different user roles. I spent the entire morning just clicking around. And the release next week already changed the memory strategy. This kind of regression was about to become a weekly nightmare. Something had to change.

The Root of the Pain: Why Manual AI Memory Testing is Pure Torture

AI chat “memory” isn’t real memory at all. It relies on the backend feeding historical conversations as context into the model. Verifying memory continuity means covering scenarios like these:

Cross-turn Context: A user says “My name is John,” and in turn 3 asks “What’s my name?” The AI must answer “John.”
Multi-user Isolation: User A’s memory must never leak into User B’s session.
Memory Window: After a certain number of turns, old memory should be pruned as expected.
Async + Streaming Responses: The AI’s reply chunks in character by character. The DOM changes constantly. Nailing the right assertion timing is a dark art.

The pain of manual testing is obvious: repetitive steps, slow execution, and easy to miss regressions. Some might say, “Just write a Selenium script!” But early Selenium had no auto-waiting; you’d end up sprinkling time.sleep(3) everywhere. When the environment slows down, the entire suite crumbles. Plus, AI chat uses streaming rendering—the last message’s CSS class might still say “generating.” How do you assert on that? The worst part is elegantly organizing “multiple turns of interaction” and “final assertion” in a script. Often, the automated script ends up more brittle than manual testing.

Approach: Playwright + pytest – Treating the Browser as a Reliable Interface

I decided to build a dedicated AI memory testing framework on top of Playwright and pytest. The core idea: Treat the entire browser session as a stateful client. pytest uses parameterization to feed in dialogue scripts, and Playwright acts them out and captures the results.

Why Playwright over Selenium?

Auto-waiting: Playwright’s locator.wait_for() and built-in auto-waiting handle streaming-rendered UIs gracefully. Assert once elements stabilize—no arbitrary sleeps needed.
Isolated Browser Contexts: Playwright’s Browser Context is like an incognito window. Different test cases share a single browser process but have isolated storage. It’s a natural fit for user isolation testing and runs much faster than spawning multiple browser instances.
Network Interception: You can capture API requests directly and assert that the messages array sent to the model indeed carries the history—truly verifying from the root whether “memory was passed along.”

Architecturally, we manage the browser and page centrally in conftest.py. Each test function receives a “clean” chat page. We use @pytest.mark.parametrize to inject different dialogue sequences. The test logic has two layers: a frontend behavior layer asserting the final AI response content, and an API contract layer checking that the request body sent to the backend correctly carries the context. With both layers combined, memory bugs have nowhere to hide.

Core Implementation: Let Code “Perform” the Multi-Turn Dialogue

The first piece of code ensures a clean session, guaranteeing tests don’t interfere with each other.

# conftest.py
import pytest
from playwright.sync_api import Page, expect

@pytest.fixture(scope="function")
def chat_page(page: Page, base_url: str):
    """
    An independent chat page for each test:
    - Clears localStorage and sessionStorage to prevent cross-case residue
    - Waits for the page to be fully interactive
    """
    # base_url is injected via the pytest-base-url plugin or command line
    page.goto(f"{base_url}/chat")
    # Wipe client-side storage to simulate a brand new user
    page.evaluate("window.localStorage.clear()")
    page.evaluate("window.sessionStorage.clear()")
    # Reload to ensure the server-side doesn't hold onto old sessions (implementation dependent)
    page.reload()
    # Wait for the chat input box to appear, confirming the page is ready
    expect(page.locator('textarea[placeholder="Type a message"]')).to_be_visible()
    return page

We use a function-scoped fixture here, running the cleanup steps per test. In practice, you can also clean up after yield. The base_url comes from the pytest command line, making it easy to switch the suite across environments.

The second piece of code is the user behavior “script,” simulating multi-turn dialogue and asserting the final memory.

# test_chat_memory.py
import pytest
from playwright.sync_api import Page

@pytest.mark.parametrize("dialogue,memory_text", [
    (
        [
            ("My name is John", None),
            ("I'm feeling great today", None),
            ("What's my name?", "John"),
        ],
        "John",
    ),
    (
        [
            ("Remember, my favorite coffee is a flat white", None),
            ("What's for dinner?", None),
            ("What is my favorite coffee?", "flat white"),
        ],
        "flat white",
    ),
])
def test_memory_continuity(chat_page: Page, dialogue, memory_text):
    """
    Executes a multi-turn dialogue and asserts the final AI response contains the key memory info.
    """
    for idx, (user_msg, _) in enumerate(dialogue):
        # Type the message and send
        chat_page.fill('textarea[placeholder="Type a message"]', user_msg)
        chat_page.click('button:has-text("Send")')

        # Only the last turn requires a strict assertion on the reply content
        if idx == len(dialogue) - 1:
            # Wait for the final AI message bubble to appear and finish loading
            # (the flag for stream rendering completion can be customized)
            last_bubble = chat_page.locator(".chat-message.ai").last
            last_bubble.wait_for(state="visible")
            # In real projects, the node changes from a "thinking" state to normal text after streaming
            chat_page.wait_for_timeout(500)  # Wait for potential DOM transitions
            # Get the full text and assert
            text = last_bubble.inner_text()
            assert memory_text in text, f"Expected to contain '{memory_text}', actual reply: {text}"
        else:
            # For non-final turns, just wait for the AI's reply to finish to avoid sending messages out of order
            chat_page.wait_for_selector(".chat-message.ai", timeout=10000)
            # A tiny delay to ensure UI stability
            chat_page.wait_for_timeout(500)

The heart of this code is the parameterized dialogue structure: a list where each element is a (user message, optional assertion). To prevent streaming rendering from messing up assertions, we locate the last AI bubble, wait for it to be visible, and then pause briefly to allow for DOM transitions—much more robust than a blind sleep(3). The dialogue can easily stretch to dozens of turns to simulate long conversations.

The third piece of code skips the UI and directly inspects the API request body to confirm memory was actually sent to the backend.

def test_memory_in_api(chat_page: Page):
    """
    Listens for /chat/completions requests and verifies the messages array contains historical context.
    """
    # List to capture requests
    api_requests = []

    # Register a listener, filtering for the target endpoint
    chat_page.on("request", lambda req: api_requests.append(req) if "/chat/completions" in req.url else None)

    # Round 1: Provide memory info
    chat_page.fill('textarea[placeholder="Type a message"]', "I love hiking")
    chat_page.click('button:has-text("Send")')
    chat_page.wait_for_selector(".chat-message.ai", timeout=15000)

    # Round 2: Question to trigger memory
    chat_page.fill('textarea[placeholder="Type a message"]', "What did I just say I love?")
    chat_page.click('button:has-text("Send")')
    chat_page.wait_for_selector(".chat-message.ai", timeout=15000)

    # Ensure the request has been sent
    chat_page.wait_for_timeout(1000)

    # Get the last /chat/completions request
    target_req = [r for r in api_requests if "/chat/completions" in r.url][-1]
    post_data = target_req.post_data_json

    # Assert that "hiking" is present in the system prompt or history within messages
    all_messages = str(post_data.get("messages", []))
    assert "hiking" in all_messages, f"API request messages should contain 'hiking', actual: {all_messages}"

    # Advanced: Check role and content integrity, e.g., system/user/assistant alternating
    roles = [m["role"] for m in post_data["messages"]]
    assert "user" in roles
    assert "assistant" in roles

This layer is the frontend test’s backup. Sometimes the UI shows the correct result, but the model didn’t actually receive the history (maybe the frontend cached it). API layer validation acts as a double lock. Playwright’s page.on("request") makes this interception trivial.

Efficiency Gains and ROI

Once the scripts were running, I split the PM’s 8 scenarios into 16 parameterized test cases and put them into CI. A single case averages about 5 seconds. 16 cases plus fixture initialization takes less than 3 minutes total. Manual testing for 8 scenarios, factoring in environment switching and screenshots, took a skilled person about 3 hours. That’s a 60x speedup in time.

More importantly, machines don’t slack off. Regression testing that used to eat 3 hours weekly now takes 3 minutes. Over a year, that’s 160+ person-hours saved. I no longer dread memory window pruning rule changes causing missed regressions. Change the strategy, run the full suite, and see the red/green lights within 5 minutes. Humans only handle the failed cases.

A bit more on ROI. Many teams treat test automation as a “when we have time” task. But AI products iterate fast, and strategies shift constantly. The cost of manual regression grows linearly or even exponentially. If you repeat the same validation process weekly, even if it’s just 2 hours, that’s 100 hours annually. This Playwright + pytest setup, from zero to covering core scenarios, takes a skilled QA about 2 days. That 2-day investment for 100+ hours saved annually offers an ROI so clear it doesn’t need recalculating.

Pitfalls and Tips

I hit some snags during implementation. Sharing them here to save everyone some time:

Streaming Response Wait Strategy: Don’t rely on fixed sleep calls. Observe the DOM structure changes. In our project, the AI reply class had a transitional state. In tests, wait for that state to disappear before asserting.
Clearing Storage Might Not Be Enough: Some apps maintain server-side sessions based on IP or User-Agent. When necessary, combine client-side clearing with API calls to end sessions or reset user states.
Async Order of Request Capture: page.on("request") is an async callback. For multiple dialogue turns in one test, make sure enough requests have landed before accessing them by index—use wait_for on a specific URL to reach an expected request count.
Parameterized Data Management: When dialogue scripts get long, putting raw JSON in decorators gets ugly. Store test data in separate JSON or YAML files and load them dynamically via pytest_generate_tests.
Token Sensitivity: You probably don’t want user-like data in your code repository. So use variables or masked placeholders for names and prompts used in assertions.

Final Thoughts

Moving from manual clicking to fully automated regression is essentially about turning “a human simulating a user” into “code simulating a user.” Playwright makes this simulation reliable enough—auto-waiting, multi-context isolation, and network interception all hit the exact sore spots of manual testing. If you’re testing an AI chat product, especially around contextual memory, multi-turn dialogue, and user isolation, definitely give the parameterized dialogue scripts + dual-layer assertion approach a try. I’ve been using it for two months, running it weekly, and it hasn’t missed a single memory-related regression bug.

When testing stops being the bottleneck, the product team can iterate on the memory experience faster. Users are getting more demanding about AI memory. One bad experience where the AI “forgets a name” can erode all trust in the product. By guarding memory with code, we’re guarding that very trust users place in our products.