I Ran 500 Playwright Tests on Our LLM Memory — 3 Deadly Bugs I Found

Last Wednesday afternoon, our business chat group exploded — “The AI clearly promised to remember my name is ‘Xiao Wang’, so why did it call me ‘User 123’ the next day?” The product manager sent screenshots, the boss @mentioned me, three messages hitting almost simultaneously. My first instinct: Was there a Redis expiration policy bug? Maybe the session didn’t realign after reconnection? That night, I decided to stop staring at logs and ran full end‑to‑end memory verification with Playwright. What I uncovered were three spine‑chilling hidden flaws.

Breaking Down the Issue: The Memory Wasn’t “Forgotten” — It Was Never Really Stored

Our scenario is a customer‑service LLM with long‑term memory. When a user says “Remember my company name is Aurora Tech,” the model should write that fact into persistent storage (we used Redis JSON with periodic disk dumps). The next time the same user asks “What’s my company name?”, it should answer correctly.

Sounds simple? Three things were sabotaging the process:

1. Serialization black hole: We directly json.dumpsed Python objects for storage and later json.loadsed them back. But we forgot to handle datetime types, causing the dump file to silently corrupt. When Redis restarted, it failed to load, and all memory was wiped to a clean slate.
2. Concurrent write race condition: If a user rapidly sent two messages like “Remember nickname A” and “Remember nickname B”, multiple workers updated the same key without locking. The last value committed was the older one.
3. Session stickiness: The memory key relied on a frontend userId stored in localStorage. If the user cleared the cache or switched devices, they immediately became a new user, and all previous memories vanished.

Unit tests rarely catch these. A mocked Redis won’t actually restart, won’t simulate browser localStorage loss, and certainly won’t fire real concurrent HTTP requests. I needed a test that could chat in a browser like a real user, then restart the entire service and verify memory recovery.

Why Playwright — Not Selenium or API Tests

API testing was ruled out immediately: the memory chain involves frontend session management (cookies/localStorage), SSE streaming responses, and browser‑side reconnection logic. Testing only the API could never reproduce the real user experience.

Selenium can also drive a browser, but Playwright’s auto‑wait, network interception, and multi‑browser context isolation let me easily simulate concurrent users without cross‑contamination. More importantly, a Playwright browserContext maps to an independent storage space (localStorage/cookies). This perfectly mimics “the same user returns after clearing cache.” I could persist one context, reuse it after restarting the service, and validate persistent storage.

So the plan became:

• Persistence test: same browserContext → set a memory → restart the service (docker compose restart) → wait for recovery → ask a question in the same context → assert the memory survives.
• Consistency test: create 10 isolated contexts → concurrently set memories → each asks its own question → assert no data mixing.

The whole suite runs on pytest + pytest-playwright, with automatic screenshots and video on failure.

Core Implementation: Let the Code Tell the Truth

1. Persistence Test: Simulate the “User Comes Back the Next Day” Journey

This snippet verifies that after setting a memory and restarting the backend, the memory is truly restored from the persistence layer.

import asyncio
import subprocess
import pytest
from playwright.async_api import async_playwright, expect

@pytest.mark.asyncio
async def test_memory_persists_after_restart():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context()  # 同一个 context 模拟同一用户
        page = await context.new_page()

        # 1. 设定记忆
        await page.goto("http://localhost:3000/chat")
        await page.fill("textarea", "请记住我的公司是极光科技")
        await page.click("button:has-text('发送')")
        # 等待模型明确回复“已记住”
        await expect(page.locator(".message.assistant").last()).to_contain_text(
            "已记住", timeout=30000  # 大模型响应慢，超时设大一些
        )

        # 2. 重启后端服务（让 Redis 重载持久化文件）
        subprocess.run(["docker", "compose", "restart", "backend"], check=True)
        # 等待健康检查通过，否则页面请求会 502
        await page.wait_for_url("http://localhost:3000/chat", timeout=60000)

        # 3. 验证记忆仍存在
        await page.fill("textarea", "我的公司叫什么？")
        await page.click("button:has-text('发送')")
        await expect(page.locator(".message.assistant").last()).to_contain_text(
            "极光科技", timeout=30000
        )

        await browser.close()

⚠️ One easily overlooked point: if the userId is stored in a cookie rather than localStorage, restarting the service won’t affect the cookie, but localStorage might depend on the server‑rendered initial state. I’ll expand on this in the pitfalls section.

2. Consistency Test: 10 Users Writing Memories Concurrently Without Overwriting

This snippet checks that when concurrent memory writes happen, different users’ memories don’t leak into one another.

import asyncio
from playwright.async_api import async_playwright, expect

async def user_set_and_check(name: str, memory: str, query: str, expected: str):