At 3 a.m., our monitoring channel exploded: users reported that the chat assistant's "memory" feature was losing data — they clicked save, but after a refresh, the memory list was blank. Ironic: all unit tests were green, CI pipelines untouched. I sat at my desk staring at the 200 response in the browser’s Network panel. The problem wasn’t code quality; we were missing end‑to‑end regression tests that truly covered what happens inside the browser. That night, we committed to moving the memory storage regression suite entirely to Playwright + pytest — and hit countless pitfalls. This article unpacks the most painful one, plus a production‑ready automation setup you can copy.
Problem breakdown
Our "memory storage" feature lets users save any snippet from a conversation. Click the memory button → frontend sends the selected summary and tags via an async request → backend generates embeddings and indexes → frontend requests the updated list and renders it.
Testing only the API or the component in isolation misses all the dirty cross‑layer interactions: a toast popping up that obscures the DOM, a stale list response arriving before the new one and overwriting it, requestIdleCallback being deferred when the tab is in the background…
Before, the team did manual regression: open the page, click the memory button, fill in a tag, save, wait for the list to refresh, click to inspect details. Covering 30+ scenarios took 2 hours, and the miss rate was sky‑high — nobody could reliably reproduce the race condition of “two rapid saves, the second one silently fails.”
We needed automation that can precisely control timing, wait for real DOM changes, and simulate cross‑session behavior. Selenium? Mixing implicit and explicit waits makes element state unpredictable. Playwright’s auto‑waiting and waitForSelector semantics look elegant, but they can still bite you hard.
Solution design
Why not Selenium?
Selenium makes you manually juggle implicit and explicit waits. When mixed, the driver waits for the implicit time first, then tries to satisfy the explicit condition — leading to stale element references everywhere. Playwright, in contrast, has actionability checks on every operation: element must be visible, stable, unobscured, large enough. It also gives you dedicated locators and frame isolation, which is a joy for SPAs. Plus, Playwright supports browser contexts natively — each test gets its own isolated cookies/localStorage, critical for cross‑session memory isolation.
Why pytest?
Fixtures are a natural fit for managing browser, page, and context lifecycles. Parametrization, markers, and parallel execution make scaling the regression suite easy. And pytest-asyncio lets you write async operations as if they were sync.
Architecture
We structured our tests in three layers:
- Infrastructure layer (conftest) – starts browsers, injects auth tokens, navigates to the memory page, and provides a core utility: “wait until the memory list rendering is stable.”
-
Business action layer – encapsulates reuseable operations like
save_memory,open_memory_detail,delete_memory. - Verification layer – strictly re‑queries elements with locators; never passes element references between steps.
With this setup, a single scenario averages 12 seconds, and 30 scenarios run in parallel in just 8 minutes.
Core implementation
1. Stable wait for memory list rendering
The trickiest part: after the backend saves successfully, the frontend must issue a new list query, get the updated array, and partially update the DOM. If you assert immediately after the API response, you’ll almost certainly grab a stale node and fail. We need a function that doesn’t just wait for network idle — it waits until the target text has actually been painted onto the page.
# conftest.py
import pytest
from playwright.sync_api import Page, expect
@pytest.fixture(scope="session")
def browser_context_args():
# 隔离存储，避免跨测试 Session 污染
return {"storage_state": None}
@pytest.fixture
def memory_page(page: Page, browser_context_args) -> Page:
# 注入 token，绕过登录
page.goto("https://app.example.com/chat")
page.evaluate("() => localStorage.setItem('token', 'test-token')")
page.goto("https://app.example.com/memory")
# 等待记忆面板可见
page.wait_for_selector("[data-testid='memory-panel']", state="visible")
return page
def wait_for_memory_item(page: Page, text: str, timeout_ms: int = 10000):
"""等待包含指定文本的记忆条目稳定出现在 DOM 中，避免 stale element"""
locator = page.locator(f"[data-testid='memory-item']:has-text('{text}')")
# 等待元素附加到 DOM 且文本稳定
locator.first.wait_for(state="attached", timeout=timeout_ms)
# 再确保内容确实是我们期望的（防止相似文本）
expect(locator.first).to_contain_text(text)
return locator
This solves the core challenge “how to reliably confirm a memory is displayed.” We first ensure the panel is loaded with wait_for_selector set to state="visible". Then we target the exact memory item using locator + has-text, wait for it to be attached so it’s not replaced mid‑assertion, and finally run expect().to_contain_text — if it fails, the error message is crystal clear.
2. Memory save and cross‑session regression test
# test_memory_storage.py
import pytest
from playwright.sync_api import expect
def test_save_and_display_memory(memory_page):
"""保存一条记忆，验证它立即出现在列表中"""
page = memory_page
# 点击对话中的记忆按钮，触发保存
page.click("[data-testid='me
(The test continues — fill the tag, confirm the save, assert the item appears using the stable wait, then open a new context to verify the memory persists across sessions.)
The key lesson? In UI automation, “waiting” is not just about timing — it’s about ensuring the right state. Playwright’s locators combined with wait_for and expect give you the precision you need, but only if you resist the temptation of shared element references and implicit globals. That 4‑hour debugging session turned into a reusable pattern that now protects every memory‑critical path we have.
Top comments (0)