Debugging LLM Conversation Memory: A 3-Day Race Condition Nightmare

Here’s what happened: We delivered an AI chat app with “memory” to a client. Every conversation turn was stored so users could pick up right where they left off. The product manager confidently said “just like ChatGPT”, and QA manually tested hundreds of rounds without issues. Then, in the second week after launch, a user complained, “The AI remembered my girlfriend’s name as my ex’s” — conversation memories were getting mixed up.

My first instinct was a database query bug. I combed through the code for an entire day — nothing. It wasn’t until I wrote an automated script with Playwright and ran a stress test in the middle of the night that I finally caught the root cause: there was an inconsistency window between the frontend’s optimistic update and the backend’s asynchronous persistence. When multiple tabs were chatting at the same time, the order of memory storage got overwritten. And manual testing simply couldn’t reproduce it reliably — human hands aren’t fast enough.

Below I’ll share the whole debugging journey, the automated verification strategy, and how we eventually fixed it. If you’re building LLM-powered apps, you’ll hit this pitfall sooner or later.

Breaking Down the Problem

Our “conversation memory” was designed like this: for every user message, the backend appends the message to a conversation history list, calls the LLM to generate a reply, and then serializes the full conversation history into Redis (key = user:{uid}:session:{sid}:messages). On the frontend, after sending a message, the new message is instantly inserted into the on‑page chat list (optimistic update), and the list is refreshed again once the backend responds.

Theoretically clear, right? But two things went wrong:

1. The Redis write was not transactional — We used SET key value to overwrite the entire JSON list. If a user sends two messages in quick succession, the LLM call for the second message might return first and SET first; then when the response for the first message comes back and SETs, it overwrites the second one.
2. Multiple tabs shared the same session_id — When a user opens two tabs, both use the same session ID. Both pages send messages, and the backend can’t distinguish which request came from which tab. This jumbles the conversation order, and the memory gets “cross‑wired”.

Conventional locking schemes are extremely awkward in this LLM scenario: LLM calls take 2–5 seconds, and you can’t just make users stare at a distributed lock spinner. What’s worse, the frontend optimistic update hides the problem even more — the UI looks perfectly normal, but as soon as you refresh you see the memory is scrambled. Manual testing could never catch this, because no one is fast enough to race-typing on two computers simultaneously.

I decided to write a deterministic reproduction script with Playwright — simulating concurrent multi‑tab chats and automatically verifying whether the memory stored in Redis matches what the page actually shows.

Designing the Verification Strategy

Why not just write backend unit tests? Because the bug is a cross‑frontend‑backend‑cache timing issue. Mocking the browser would mask the real race‑condition window. We needed to actually launch Chromium, let two pages operate on the same session concurrently, and then inspect Redis’s actual storage after each step.

Tech stack:

• Playwright (Python): Simulates the browser, supports concurrent contexts (multi‑page) natively, and allows precise timing control.
• pytest: Orchestrates the test cases, with the pytest-playwright plugin managing the browser lifecycle.
• redis-py: Reads Redis directly inside the test to compare the stored conversation against the expected order.
• Not Selenium: Playwright’s page.wait_for_selector and network waiting mechanisms are more reliable, and its built‑in concurrent contexts make simulating multiple tabs natural.

Architecture: the test script launches two independent browser contexts (simulating two tabs) logged in with the same user and session. Then “Tab A” sends message m1, and without waiting for the reply, “Tab B” immediately sends message m2. At four critical time points — after sending m1, after m1’s reply arrives, after sending m2, and after m2’s reply arrives — we read the memory snapshot from Redis. The final assertion: the message list in Redis must neither lose any message nor scramble the order (m1 must appear before m2).

Core Implementation

1. Simulating concurrent chats and capturing snapshots

This snippet demonstrates how to manufacture a race condition with deterministic timing using Playwright while grabbing Redis snapshots. We use two async tasks to simulate the two tabs, precisely coordinating the sending moments via asyncio.Event.

import asyncio
import json
import redis
from playwright.async_api import async_playwright

REDIS_CLIENT = redis.Redis(host='localhost', port=6379, decode_responses=True)

def get_memory_snapshot(user_id: str, session_id: str) -> list:
    """从 Redis 读出当前存储的对话记忆（已解析的消息列表）"""
    key = f"user:{user_id}:session:{session_id}:messages"
    raw = REDIS_CLIENT.get(key)
    return json.loads(raw) if raw else []

async def send_message(page, text: str):
    """发送一条消息并等待 AI 回复出现在页面上"""
    await page.fill('textarea[placeholder="输入消息"]', text)
    await page.click('button[type="submit"]')
    # 等待 AI 回复的容器出现且包含文本，避免拿到空壳
    await page.wait_for_selector('.message.assistant:last-of-type :text-is("")', state='detached', timeout=15000)
    await page.wait_for_selector('.message.assistant:last-of-type', state='visible')

async def concurrent_chat_test():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        # 创建两个独立 context，模拟两个标签页，共享同一 session
        context = await browser.new_context()
        page_a = await context.new_page()
        page_b = await context.new_page()

        USER_ID = "test_user_01"
        SESSION_ID = "sess_concurrent_01"
        # 省略登录逻辑，直接注入 localStorage/jwt
        await page_a.goto(f"https://chat.example.com/chat?session=