It was 1 a.m. I had just closed my laptop, ready to call it a night, when the testing group chat suddenly exploded. Our QA teammate tagged me: “Your customer service agent forgot everything again. I told it I was a VIP in the last turn, and now it’s asking me to re-verify my identity. Users are going to lose it.” I checked the logs—Redis had the session context written correctly—but the agent simply wasn’t recalling it. After manually reproducing the issue seven or eight times, with it working intermittently, I finally traced the root cause to a timezone mismatch that messed up the TTL calculation. The memory was expiring the moment it was written. The fix took exactly one line of code, but the debugging marathon convinced me of one thing: memory validation has got to be automated.
Why AI agent memory validation is such a pain
Our agent is a conversational system with both short-term and long-term memory. Short-term memory relies on Redis to cache the last N dialogue turns; long-term memory uses a vector database for semantic recall—things like user preferences and historical tickets. The product logic is straightforward: when a user says, “I wasn’t satisfied with plan A you recommended last time,” the agent must recall the recommendation record from three days ago and offer plan B.
Manual validation has very specific pain points:
- Dialogue path explosion: Verifying memory requires multi-turn conversations. A single path like “greeting → make a request → add conditions → switch topic → recall old memory” takes at least five minutes to click through by hand.
- Persistence is a black box: Testers can’t see whether short-term memory actually landed in Redis, whether long-term memory was inserted into the vector database, or whether the TTL is set correctly.
- Strict recall conditions: Sometimes memory isn’t actually lost—it’s just invisible because the similarity threshold is too high, or the prompt formatting is off, causing the LLM to “not see” that memory. Regression testing means rerunning a dozen cases, which is simply unbearable to do manually.
Conventional API automation (like pytest + requests) can verify the storage layer but can’t cover the full end-to-end link: “user input → memory storage → recall in a new conversation → the LLM uses it correctly.” On top of that, memory is often tied to a session, so API tests have to maintain their own cookie/session state—making the test code more complicated than the business logic itself.
Solution design: why Playwright
What I needed was a tool that could operate the browser like a real user, be completely non-intrusive, and let me manipulate browser state at will.
A quick comparison of the alternatives:
- Selenium: mature ecosystem, but asynchronous waiting for AI replies is clumsy and frequently runs into timeouts.
- Puppeteer: JavaScript ecosystem, but our backend is mostly Python, so the maintenance cost for test code is higher.
- Cypress: better suited for front-end unit/component testing; multi-tab and cross-origin scenarios are limited.
I chose Playwright because it natively supports async/await, has built-in auto-waiting, can intercept network requests (to verify that the API actually calls the memory storage endpoint), and—most importantly—can save and restore browser state (storageState). This perfectly simulates the scenario of “closing the browser and coming back, with the session memory still intact.”
The architecture is simple: a set of Python test scripts uses Playwright to launch Chromium and interact with our agent’s front-end chat page. Each test case walks through a complete “memory write → sleep → memory recall” loop. Assertions don’t rely on fragile string matches in the UI; instead, I combine them with API intercepts to confirm that the memory service request body includes the expected context.
Core implementation: building a reliable memory test step by step
Install dependencies first (no code needed, but you must get this running):
pip install pytest-playwright pytest-asyncio
playwright install chromium
1. Using storageState to simulate “close and reopen without losing memory”
This snippet solves the problem of verifying whether short-term memory persists across sessions. The core idea is to simulate closing the tab and reopening it while keeping the session alive.
# test_memory_persistence.py
import pytest
from playwright.async_api import async_playwright
import asyncio
BROWSER_STATE_PATH = "state.json" # 持久化storageState
async def test_short_term_memory_persistence():
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True) # CI环境也可跑
# 第一个会话：建立记忆
context = await browser.new_context()
page = await context.new_page()
await page.goto("http://localhost:3000/chat")
await page.fill("input#message", "我叫张三，我是白金会员")
await page.click("button#send")
# 等待AI回复（auto-wait会等网络空闲，但不保险，所以加一句）
await page.wait_for_selector("div.assistant-message", timeout=10000)
# 保存浏览器状态，包括cookie/localStorage（session token在里面）
await context.storage_state(path=BROWSER_STATE_PATH)
await browser.close()
# 第二个会话：恢复状态，验证记忆还在
browser2 = await p.chromium.launch()
context2 = await browser2.new_context(storage_state=BROWSER_STATE_PATH)
page2 = await context2.new_page()
await page2.goto("http://localhost:3000/chat")
await page2.fill("input#message", "我的会员等级是什么？")
await page2.click("button#send")
# 关键断言：回复中应包含“白金”
response_text = await page2.text_content("div.assistant-message:last-child")
assert "白金" in response_text, f"记忆丢失，实际回复: {response_text}"
await browser2.close()
Why write it this way? storage_state saves cookies and localStorage as-is and injects them into the new context. The server sees the exact same session ID, which allows us to directly verify whether the short-term memory in Redis is still within its TTL and can be recalled. Previously, manual testing often...
Top comments (0)