At 1 a.m., I was jolted awake by a barrage of DingTalk messages. Our product manager shouted in the group chat: “Users are complaining that our AI assistant completely forgot the ‘emergency contact’ they set up last week, even though it was mentioned in a previous conversation!” Groggily, I opened the logs and scrolled through the chat history. The model’s latest response indeed did not match the historical memory—a classic memory hallucination. But the previous version had worked fine. Who had changed the memory storage logic? No one admitted it. That night I manually compared 30 conversation threads until my eyes glazed over, then made a gut decision: rollback. Later I found myself wondering, why, in 2025, are we still relying on “manual eyeballing” to verify large model memory? That question directly led to our automated regression approach using Playwright + VCR.
The Problem: Why Testing Large Model Memory Is So Hard
Our product has a “long‑term memory” feature: the model remembers a user’s personalised information (name, preferences, recently discussed projects) across multi‑turn conversations and recalls it later. The memory storage layer is built on a vector database and a summarisation engine. Every time we upgrade the memory extraction logic, swap the embedding model, or tweak a prompt, we risk a regression—either missing memory entries or generating completely wrong new memories (hallucination). The manual regression flow was: open the chat window, play the user role by following a scripted conversation, then manually verify that the model remembered what it should and did not fabricate anything. This flow had three fatal pain points:
- Time‑consuming and unreliable: Running a 20‑turn script took a skilled tester 15 minutes. By the fourth repetition fatigue kicked in, and the false‑positive rate shot up when comparing similar dialogues. Our product manager once marked a hallucinated answer as “fine” because it “looked reasonable enough.”
- Narrow coverage: We could only spot‑check typical paths. Changing a single line of memory deduplication code left us completely unsure whether edge‑case long conversations were affected.
- Environment differences: Something that worked perfectly locally might behave differently in staging due to different model versions or API latency. Manual testing could never cover that gap.
What about conventional test approaches? Unit tests that mock the LLM API only verify the storage layer’s code logic—they completely miss the model’s real unpredictability, the timing of frontend interactions, and the exact moment streaming triggers memory extraction. Selenium scripts can control the browser, but asserting memory content requires a pile of sleep calls and fragile selectors; the maintenance cost ends up higher than the system under test. What we needed was a method that worked like a video replay: freeze the entire user–AI interaction as a baseline, then automatically compare every future run—any deviation is a bug.
Solution Design: Why Playwright + VCR Mode
The selection logic was straightforward: use Playwright to drive a real browser and simulate user input, clicks, and waiting for streaming responses. At the same time, borrow the VCR (Video Cassette Recorder) metaphor: record the “conversation sequence + AI response text + key DOM snapshots” of each test run. The first run is set to record mode, producing a baseline. Subsequent regression runs automatically enter replay mode and perform a turn‑by‑turn comparison with the baseline. Any text difference, missing memory, or hallucination contamination gets caught immediately.
Why not the alternatives?
- Cypress: Its support for multi‑tab and WebSocket streaming updates isn’t as native as Playwright’s, and our chat interface uses SSE streaming.
- pytest-vcr (HTTP mocking): It records and replays LLM API calls, which saves tokens, but also means you are forever testing against the same stale model responses—you’d never catch new hallucinations introduced by a model upgrade or a memory prompt change.
- Pure screenshot comparison (visual regression): AI‑generated text length varies wildly; a single scroll offset can turn the entire pixel grid red and drive your false‑positive rate through the roof. We therefore compare only structured text and the existence of certain CSS classes.
Our final architecture: pytest + Playwright synchronous API + a lightweight custom VCR tool we named ChatVCR. It serialises each conversational turn (user input, expected memory assertion, actual AI response) into JSON. Baselines are stored under tests/cassettes/, with paths generated automatically from test function names. A --vcr-record CLI flag controls recording vs. replaying.
Core Implementation: A Truly Usable ChatVCR Test Framework
The code below solves one core problem: how to let a Playwright test script automatically record conversation baselines, then compare them precisely during regression while avoiding interference from random text (timestamps, IDs).
1. The ChatVCR base class – responsible for recording and comparison
This snippet defines the ChatVCR class, wrapping conversation save/load and smart comparison logic. The key is that during comparison a strip_dynamic_text function uses regular expressions to clean dynamic content that changes across runs—otherwise the baseline would never match.
# chat_vcr.py
import json
import re
from pathlib import Path
from typing import Any, Dict, List
class ChatVCR:
def __init__(self, cassette_path: str, record_mode: bool):
self.path = Path(cassette_path)
self.record_mode = record_mode
self.dialogues: List[Dict[str, Any]] = []
def add_turn(self, user_msg: str, assistant_response: str, expected_memory: str = ""):
"""录制一轮对话"""
self.dialogues.append({
"user": user_msg,
"assistant": self._clean(assistant_response),
"expected_memory": expected_memory
})
def save(self):
if self.record_mode:
self.path.parent.mkdir(parents=True, exist_ok=True)
self.path.write_text(json.dumps(self.dialogues, ensure_ascii=False, indent=2))
def load_expected(self) -> List[Dict[str, Any]]:
if not self.path.exists():
raise FileNotFoundError(f"Baseline not found at {self.path}, run with --vcr-record first")
return json.loads(self.path.read_text())
de
Top comments (0)