At 2 AM, my testing colleague’s WeChat voice call jolted me awake: “The conversation memory bug that drops context is back. I ran a full regression and still couldn’t capture a stable reproduction path.” Rubbing my eyes, I opened my laptop, manually launched five browser windows, typed “My name is Ming, remember that,” refreshed the page, asked “What’s my name?” and ticked off results against screenshots. That night I spent nearly two hours just to check whether a memory persistence feature had been broken again by a backend change. The thought that burned in my mind: Manually verifying memory in a browser is like mining with a hand shovel – painfully inefficient and borderline absurd.
Then I slammed the desk and wired up Playwright with LangChain, making the browser automatically chat, refresh, and ask questions, then letting an LLM decide “did it remember or not.” Regression time dropped from 2 hours to 3 minutes, while we managed to cover 30+ edge cases in the same run. This post shares the idea and complete code behind this “automated memory bodyguard.”
Breaking It Down: Why Manual Memory Persistence Testing Is a Nightmare
The essence of “conversation memory persistence” in LLM applications is this: after the frontend is closed or refreshed, the session history and knowledge base remain alive. The standard verification path is:
- Open the web page, chat with the bot for a few rounds, inject personal info (name, preferences, to-dos).
- Close or refresh the page.
- Send another message and visually check whether the response references the earlier information.
Running this manually takes at least 2–3 minutes per cycle. If the system under test uses a deeply customized LangChain Memory, a custom Redis store, or a conversation flow with complex routing, regression needs to cover 20+ conversation shapes (multi-turn, single-turn, mid-disconnect, memory after token trimming, etc.). Humans simply can’t keep up, and “eyeballing” whether the bot remembered is terribly unreliable – sometimes the bot just politely says “Hello,” and you have no idea if that reply came from memory or was randomly generated.
The usual automation approaches are threefold:
- Direct API calls: bypasses the browser and can’t verify whether frontend cookies / localStorage / session data synced correctly to the backend.
- Selenium + fixed assertions: asserting “response contains ‘Ming’” misses tons of natural variations – the model might say “Your name is Ming, right?” Fixed pattern matching is practically blind.
- Fully manual: slow, unrepeatable, error-prone.
This job requires a semantic judge that truly understands meaning, plus a robot that can drive a browser.
Solution Design: Playwright Drives the Browser, LangChain Judges
I raided my toolbox:
Why Playwright instead of Selenium?
Playwright’s auto-wait is far friendlier to modern SPAs – no more false timeouts when an element is in the DOM but not yet rendered. It also effortlessly manages multiple browser contexts, perfect for simulating “refresh and re-enter.” And its Python API mixes seamlessly with LangChain.
Why LangChain for assertions?
Asserting “memory retained or not” is fundamentally a semantic judgment task. Instead of writing a hundred regular expressions, just hand it to ChatOpenAI (or any LLM) and let it respond “yes/no.” LangChain’s ChatModels and Prompt templates let you quickly build a reusable evaluator, and you can easily swap models for cost control.
Alternatives I deliberately skipped:
- Pure LLM API to compare texts: no coverage of the frontend link.
- Browser extension recording & playback: no intelligent assertions; any minor DOM change breaks everything.
- Robot Framework + custom keywords: high extension overhead; just using Python directly is more flexible.
The final architecture is dead simple: Playwright script → browser simulates conversation & refresh → grabs the last reply → LangChain judge => pass/fail, then collect results into a CLI report.
Core Implementation: Three Building Blocks That Form the Memory Bodyguard
The runnable Python code below is split into three functions, tackling simulated conversation, triggering the memory check flow, and intelligent assertions respectively.
1. Playwright Simulated Conversation – Chat the Bot into Position
This code automatically inputs messages, clicks send, and waits for the assistant’s reply to appear. All selectors use common semantic placeholders; tweak them for your project and you’re good to go.
from playwright.sync_api import sync_playwright
def start_session_and_chat(page, initial_messages: list[str]):
"""
page is a Playwright Page object already on the chat page,
initial_messages is a list of strings to send in order.
"""
for msg in initial_messages:
# Wait for input field to be interactive (SPA loading friendly)
page.wait_for_selector("textarea[placeholder='输入消息...']", state="visible")
page.fill("textarea[placeholder='输入消息...']", msg)
page.click("button:has-text('发送')")
# Wait for at least one assistant message to appear before continuing, to avoid overlap
page.wait_for_selector(".assistant-message:last-child", timeout=10000)
# Usage example: implant memory
with sync_playwright() as pw:
browser = pw.chromium.launch()
page = browser.new_page()
page.goto("http://localhost:3000/chat")
start_session_and_chat(page, ["你好", "我叫小明，我最喜欢蓝色的跑鞋"])
The wait_for_selector and .assistant-message:last-child are key for handling asynchronous streaming replies. If your UI uses SSE, you might need additional DOM change listeners, but usually waiting for the latest message node is enough.
2. Memory Persistence Verification Flow – “Quiz” It after Refresh
We deliberately inject a set of personal information, then refresh the page to simulate a disconnect, and ask a targeted question.
python
def verify_memory_after_reload(page, user_name: str, memory_keyword: str) -> bool:
"""
Implant name and preference into the bot, refresh, then ask whether it remembers.
user_name : user name, used for questions like "What's my name?"
memory_keyword : expected keyword in the response (e.g., the name).
"""
# Implant phase
start_session_and_chat(page, [
f"记住：我
Top comments (0)