The 6-Hour Playwright Bug: How "Element Visible" Broke My LLM Long-Term Memory Tests

At 1 a.m., the QA group blew up my chat: “The memory feature is broken! Everything we told it yesterday is gone!” I rolled over, thinking I'd have to manually click through a dozen conversations again. The next morning, with dark circles under my eyes, I decided to script the whole thing so I’d never have to do it manually again. The moment the script was ready, I naively thought I was done — until CI lit up red with failures screaming “element is not interactable.” I spent the entire weekend staring at Playwright traces for six hours before finally catching the culprit.

This article captures that blood, sweat, and debug tears, and gives you a fully working script that you can run right now.

---

The Problem: What You’re Testing and Why Manual Checks Are a Nightmare

LLM “long-term memory” isn’t magic — it essentially vectorizes past conversations and injects the relevant bits into the prompt during inference. To verify that memory recall works, a typical flow looks like this:

1. In a session, tell the bot “My favorite color is Prussian blue.”
2. Log out, wait some time (or switch devices), and log back in.
3. Start a new conversation and ask, “What did I say my favorite color was?”
4. Expect the answer: “Prussian blue.”

Manual validation looks simple, but the pain points pile up quickly:

• Multiple interaction rounds plus login/logout — a single run takes at least 3–5 minutes.
• Recall can be affected by time decay or RAG strategy changes — you need to cover different time gaps and question variants.
• LLM responses are inherently non‑deterministic — the same question can produce five equally correct answers, making assertions fragile.
• You can’t ask QA to manually click through 20 times every regression cycle.

So we desperately need a fully automated, repeatable test suite that gracefully handles LLM output uncertainty.

---

Solution Design: Why Playwright + Explicit State Isolation

The job is straightforward: simulate real user behavior with browser automation. The tech choices boiled down to two major options:

• Selenium – mature and stable, but slow startup and multi‑session isolation requires special WebDriver configurations.
• Playwright – natively supports multiple BrowserContexts, each with its own storage (cookies/localStorage), offering excellent isolation; its modern waiting mechanisms can listen for network idle, stable elements, and more.

I picked Playwright for two core reasons:

1. Multi‑context isolation fits the memory‑verification scenario perfectly — you can use one browser instance, one context for “training,” and a fresh context for recall validation, guaranteeing session‑level isolation without simulating two full logins.
2. Its expect assertions come with built‑in retries and timeout tolerance — much more reliable than manual sleep() when LLM response times fluctuate wildly.

Architecture: one script, three phases

• Setup phase – Log in → send “training information” → wait for LLM confirmation that it’s stored.
• Train phase (optional) – Continue multi‑turn conversation to reinforce the memory.
• Recall phase – Create a new context → log in with the same account → ask a “recall” question → assert the response contains the target entity.

Why not just test within the same context? Real‑user long‑term memory is cross‑session. If you don’t clear storage, the test will often “false‑pass” — the LLM may simply copy the information from the current conversation history rather than performing a true vector recall.

---

Core Implementation: A Production‑Ready Playwright Test Script

1. Build the test skeleton and a login utility

This snippet shows how to quickly open the app and log in with Playwright. I’m using playwright.sync_api for readability; you can switch to the async version if your project requires it.

# test_longterm_memory.py
import pytest
from playwright.sync_api import Playwright, BrowserContext, Page, expect

APP_URL = "https://your-llm-chat.example.com"
USER_EMAIL = "[e-mail]"
USER_PASS = "securepassword"

@pytest.fixture(scope="session")
def browser_context(browser: Browser) -> BrowserContext:
    # Create an isolated browser context – a fresh user environment
    context = browser.new_context()
    yield context
    context.close()

def login(page: Page):
    page.goto(f"{APP_URL}/login")
    page.fill('[data-testid="email-input"]', USER_EMAIL)
    page.fill('[data-testid="password-input"]', USER_PASS)
    page.click('[data-testid="login-button"]')
    # Wait for the post‑login redirect to the chat main page
    page.wait_for_url(f"{APP_URL}/chat", timeout=10000)

2. Train the memory: feed personal info and confirm reception

This part addresses “how to reliably feed in memory and capture confirmation.” The key is to wait until the LLM’s streaming response has finished before asserting, otherwise the DOM might still be mutating.

def train_memory(page: Page, nickname: str, favorite_color: str):
    """Input personal information in the chat and confirm the bot received it."""
    page.goto(f"{APP_URL}/chat")
    # Assume there is a "new chat" button
    page.click('[data-testid="new-chat-button"]')
    # Type the message
    input_box = page.locator('[data-testid="message-input"]')
    input_box.fill(f"我叫{nickname}，我最喜欢的颜色是{favorite_color}。记住这个。")
    page.click('[data-testid="send-button"]')

    # Critical point: wait for the LLM response to finish – not just a selector, but network idle
    # This is the first setup for a major pitfall
    page.wait_for_load_state("networkidle")

    # Loose assertion: only check that the reply contains a storage‑confirmation keyword
    response = page.locator('[data-testid="assistant-message"]').last
    expect(response).to_contain_text("记住", timeout=15000)  # LLM may say "我记住了" or "已存储"

3. Verify recall: new context, same login, ask and assert

Now comes the recall test. We create a brand‑new BrowserContext (simulating a new session/device), log in with the same credentials, and ask a question that should trigger the stored memory. The assertion must be flexible enough to match “Prussian blue” in various natural responses.

def test_longterm_memory_recall(browser: Browser, browser_context: BrowserContext):
    # Phase 1: Setup – train the memory in the first context
    page1 = browser_context.new_page()
    login(page1)
    train_memory(page1, nickname="小明", favorite_color="普鲁士蓝")
    page1.close()

    # Phase 2: Recall – brand new context, no shared storage
    recall_context = browser.new_context()
    page2 = recall_context.new_page()
    login(page2)

    page2.goto(f"{APP_URL}/chat")
    page2.click('[data-testid="new-chat-button"]')
    question = "我之前说过我喜欢什么颜色？"
    page2.fill('[data-testid="message-input"]', question)
    page2.click('[data-testid="send-button"]')

    # Wait for the full streaming response
    page2.wait_for_load_state("networkidle")

    response = page2.locator('[data-testid="assistant-message"]').last
    # Flexible assertion: the response should contain the exact color name
    # This handles variations like "你之前提到你喜欢的颜色是普鲁士蓝" or "我记得是普鲁士蓝"
    expect(response).to_contain_text("普鲁士蓝", timeout=15000)

    page2.close()
    recall_context.close()

---

The Six‑Hour Debug: When “Element Visible” Lies

The script above looks clean, right? I ran it locally — green. Pushed to CI — a wall of red. Every failure screamed element is not interactable or click intercepted. But clicking and typing worked fine when I stepped through with --headed locally. The CI logs were no help; the screenshots showed a fully loaded chat UI. So what’s the deal?

The breakthrough came when I fired up Playwright Trace Viewer. The trace showed that after the fill and click actions, the page performed a partial re‑render triggered by the LLM’s streaming animation. The “send” button was technically visible in the DOM and passed visible checks, but a translucent overlay (a streaming indicator or skeleton loader) was briefly blocking it. Playwright’s actionability checks rightfully refused to interact — the element was visible, but not receiving events.

To make it worse, the overlay appeared and disappeared so fast that the naked eye couldn’t catch it — only the trace’s millisecond‑level timeline exposed it.

The fix: wait for the right state, not just visibility

Instead of wait_for_load_state("networkidle") alone (which waits for network activity, not necessarily UI stability), I added explicit waits for the overlay to disappear and for the input element to become truly interactable.

# After sending a message, wait for the streaming overlay to vanish
page.locator('[data-testid="streaming-indicator"]').wait_for(state="hidden", timeout=15000)
# Then explicitly wait for the input box to be editable
expect(input_box).to_be_editable(timeout=5000)

Even better, in many cases Playwright’s fill/click already perform actionability checks. The real lesson is to never assume that visible equals interactable in highly dynamic UIs. The LLM streaming layer can (and will) insert fleeting overlays that escape casual inspection.

---

Why This Pattern Works for LLM Memory Tests

• Session isolation mimics real cross‑device memory recall, catching bugs where memory persists only within the same browser session.
• Fuzzy assertions with to_contain_text tolerate LLM verbosity without false negatives.
• Explicit state waits handle streaming UI quirks that would otherwise cause frustratingly flaky tests.
• The whole suite runs in headless mode on CI and costs seconds per scenario.

---

Takeaways and the Full Script for You

If you’re testing LLM memory features, here’s my hard‑earned advice:

1. Isolate contexts — otherwise you’re testing chat history, not long‑term memory.
2. Trace early, trace often — the bug you can’t see in screenshots is waiting in the timeline.
3. Assume “visible” is a lie — always wait for the exact state that signals full UI readiness (overlays gone, input editable).
4. Embrace fuzzy assertions — an LLM that answers with 100% deterministic strings doesn’t exist; your tests shouldn’t pretend otherwise.

The complete working script is above. Drop it into your repo, adjust the selectors, and may your overnight CI runs stay green. And if you ever find yourself fighting a flaky Playwright test at 2 a.m., remember: it’s probably a ghost overlay. Check the trace.