DEV Community: BAOFUFAN

How We Caught 90% of Our LLM Memory Bugs with Playwright

BAOFUFAN — Wed, 01 Jul 2026 12:04:23 +0000

At 1 a.m., our product manager dropped a screenshot in the group chat: a user asked, “Repeat the order number I just gave you,” and our AI customer-service bot replied, “Please provide your order number.” The user cursed three times, and the screenshot ended up on a complaint platform.

This wasn’t a model capability problem—the model had simply “forgotten” the context. Even worse, I manually tested the same conversation three times and couldn’t reproduce it once. Then I wrote 200 automated memory tests using Playwright, and 90% of the forgetfulness pointed to one embarrassingly basic problem: inconsistent serialization and deserialization in the front-end storage layer.

Breaking down the problem: how memory silently disappears

Our customer-service bot is built on an LLM with LangChain, and the front end uses a React chat component. Conversation history is kept in the browser’s localStorage. Every time a user refreshes or reopens the page, the front end reads the history from localStorage and stitches it into the prompt to maintain long-context memory.

Sounds great in theory. In reality, many users access the service from mobile browsers, and the page gets killed frequently by the OS. When the session restores, we rely on that locally cached conversation to rebuild memory. If anything goes wrong during storage or retrieval, the model suffers amnesia.

Why didn’t manual testing catch it? Because we unconsciously test with the “correct” sequence—send message A, then message B, then ask about message A’s content. Real users, however, may refresh the page, switch apps, or even have their phone die and restart. The effect of these actions on the front-end storage is almost impossible to reproduce manually. Traditional unit tests only cover JS logic; they can’t test the complete “browser-level persistence and restoration” path.

Why Playwright for end-to-end memory verification?

I needed a tool that could control a real browser, manipulate localStorage, intercept network requests, and assert page content. Selenium is too heavy, Puppeteer only supports JS, Cypress is intrusive and has weak multi-tab support. Playwright supports Python, has a clean API, and can directly push arbitrary content into localStorage via page.evaluate(). It also simulates page refreshes and cross-page restores perfectly.

Architecturally, I divided the tests into three layers:

Scenario layer: defines memory test scenarios, e.g., “order number recallable after multiple conversation turns,” “memory not lost after refresh.”
Driver layer: wraps Playwright operations and provides high-level actions like send_message and reload_and_restore.
Verification layer: asserts memory consistency from two dimensions—page text and localStorage contents.

This way, even if the front-end framework changes from React to Vue, I only need to update selectors; the test logic stays intact.

Core implementation: let Playwright repeat every bizarre user action

1. Setting up a repeatable test environment

We need a runnable chat page. I used a small Streamlit app as a stand-in (launched with subprocess and killed after tests to ensure isolation). You can replace it with any front-end page.

import subprocess
import time
import pytest
from playwright.sync_api import sync_playwright

# 启动一个本地 Streamlit 聊天应用，模拟 LLM 对话服务
def start_chat_server():
    proc = subprocess.Popen(
        ["streamlit", "run", "chat_ui.py", "--server.port", "8510"],
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL
    )
    time.sleep(3)  # 等待服务就绪
    return proc

@pytest.fixture(scope="module")
def page():
    server = start_chat_server()
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        context = browser.new_context()
        page = context.new_page()
        page.goto("http://localhost:8510")
        yield page
        browser.close()
        server.terminate()

This snippet solves “how to guarantee each test starts from a clean state.” new_context() ensures isolated storage space, preventing cross-test contamination.

2. Simulating multi-turn conversation and asserting memory retention

The test below is the core: the user first provides an order number, chats about unrelated topics, then asks the bot to repeat the number. We expect the model to correctly recall it.

def test_order_id_recall_after_conversation(page):
    # 封装发送消息的函数，处理前端可能的异步渲染
    def send_message(text: str):
        # 找到输入框并键入消息。实际选择器按你的前端结构调整
        page.fill('textarea[aria-label="输入消息"]', text)
        page.click('button:has-text("发送")')
        # 等待 AI 回复出现（以最后一条气泡里的文本为准）
        page.wait_for_selector(".message.assistant:last-of-type .content", timeout=5000)

    # 用户提供订单号
    send_message("我的订单号是 ORD-9876，查一下物流")
    # 中间几轮无关对话，目的是扩大上下文距离，更容易触达遗忘边界
    send_message("这个商品支持七天无理由吗？")
    send_message("发货一般要多久？")
    # 关键轮次：再次询问订单号
    send_message("我刚才说的订单号是什么？你重复一遍")

    # 获取最后一条助手的回复文本
    last_reply = page.text_content(".message.assistant:last-of-type .content")
    # 核心断言：回复中必须包含之前给出的订单号
    assert "ORD-9876" in last_reply, (
        f"记忆丢失！模型没有复述订单号，实际回复: {last_reply}"
    )

If this step fails, it’s classic long-context forgetting—something manual testing would likely never hit because nobody manually pads exactly the right number of distraction turns.

3. Testing memory recovery after refresh — the real danger zone

When a user refreshes the page or restores it from the background, the front end reads from localStorage to rebuild memory. Common serialization mistakes (e.g., storing and reading bigint, Date objects) cause the restored conversation to be truncated or empty, leading to immediate amnesia.

def test_memory_persists_after_reload(page):
    def send_message(text: str, sender="user"):
        # 这个测试中我们直接操作 localStorage 模拟前置对话
        # 更贴近“用户之前聊过，现在刷新页面回来”的场景
        pass

In the actual test script I’ll first inject a complete conversation into localStorage, then reload the page, and send a new message referencing the conversation. If the LLM can’t recall, the serialization is broken. The code below demonstrates how to inject and verify.

def test_memory_after_reload_and_send(page):
    # 直接往 localStorage 里写入一段历史对话（模拟用户之前聊过）
    existing_conversation = [
        {"role": "user", "content": "我的订单号是 ORD-1234"},
        {"role": "assistant", "content": "好的，我记下了您的订单号 ORD-1234。"}
    ]
    page.evaluate(
        "window.localStorage.setItem('chat_history', JSON.stringify(arguments[0]))",
        existing_conversation
    )

    # 刷新页面，模拟用户重新打开或刷新
    page.reload()
    # 等待页面加载完成
    page.wait_for_selector('textarea[aria-label="输入消息"]', timeout=5000)

    # 续接对话：询问之前告知的订单号
    page.fill('textarea[aria-label="输入消息"]', "我之前说的订单号是什么？")
    page.click('button:has-text("发送")')
    page.wait_for_selector(".message.assistant:last-of-type .content", timeout=5000)

    last_reply = page.text_content(".message.assistant:last-of-type .content")
    assert "ORD-1234" in last_reply, f"刷新后记忆丢失，回复内容: {last_reply}"

4. Verifying `localStorage` contents directly — catching hidden corruption

Sometimes the model still works in isolated tests but the stored data is silently mangled. By adding a direct assertion on the localStorage string, we can detect inconsistencies early.

def test_localstorage_integrity(page):
    page.fill('textarea[aria-label="输入消息"]', "我的会员ID是 VIP-2024")
    page.click('button:has-text("发送")')
    page.wait_for_selector(".message.assistant:last-of-type .content", timeout=5000)

    raw = page.evaluate("window.localStorage.getItem('chat_history')")
    assert raw is not None, "localStorage 中没有对话历史"
    parsed = json.loads(raw)
    # 验证关键数据没有被序列化破坏
    assert any("VIP-2024" in msg.get("content", "") for msg in parsed), (
        "localStorage 中的数据丢失了原始信息"
    )

This test alone caught 8 different serialization bugs during the sprint.

The real culprit: serialization silently breaks your context

Most people treat JSON.stringify and JSON.parse as transparent. They aren’t, especially when custom objects, undefined, or non-serializable values sneak into the conversation array. In our case:

A Date object from a message timestamp was directly stored, and after parsing it became a string, causing downstream code to skip that message because the valid-time check failed.
Some messages had content fields set to undefined (due to streaming glitches). JSON.stringify converts undefined to null, but the restoration logic expected typeof checks on undefined, causing an entire dialog block to disappear.
On mobile browsers with tight storage quotas, partial writes happened, leaving the JSON structure broken.

After fixing all these, the memory test pass rate went from 32% to 96% overnight.

Takeaways: make memory testing a habit, not an afterthought

Test memory logic in the browser, not in unit tests. Serialization is a runtime concern; unit tests with mocked localStorage won’t catch real-world quirks.
Write adversarial memory scenarios. Refresh, background-kill, and restore from partial state—these are exactly Playwright’s strengths.
Verify at multiple layers. Page output and raw localStorage content must align.
Automate relentlessly. If you have an AI feature that depends on context, you should have hundreds of Playwright tests that wear down the context like a user with a habit of switching apps.

That 1 a.m. bug report was a gift. It pushed me to stop trusting manual memory tests and start treating front-end storage as a first-class citizen. Playwright gave me the power to simulate the chaotic behavior of real users, and those 200 tests now run on every CI build. Memory bugs? They show up in 90 seconds now, not in customer complaints.

If you’re struggling with similar long-context memory issues, consider adding a “memory canary” test—a simple prompt-response pair that must survive whatever your front-end storage does. It might be the most productive 20 lines of code you write this week.

Chasing Chroma’s Silent Data Loss: How Playwright E2E Tests Saved 72 Hours

BAOFUFAN — Wed, 01 Jul 2026 01:03:06 +0000

At 2 AM, a message exploded in our user group: “Why does the AI assistant ask my name again? It remembered I’m Xiao Li yesterday.” I checked monitoring — memory normal, CPU stable, no errors in logs. After restarting the service, the memory came back — only to disappear again the next day. No error, no exception stack trace. Chroma had silently dropped the documents. That was when I realized: conventional unit tests are useless when it comes to vector databases.

Breaking down the problem

This LLM app uses a typical RAG memory pipeline: user conversations are summarized and written into the vector store; later queries retrieve relevant memories to inject into prompts. Memory persistence relies on Chroma’s persist(). In integration tests we mocked its response, wrote a few documents, queried them — all green. But production is a different story.

The root cause is a combination of three factors. First, Chroma’s default persist() does not flush to disk immediately; the underlying SQLite has its own caching strategy, so if the process exits unexpectedly, the last few seconds of writes can vanish. Second, we used async add_texts but returned HTTP 200 before the underlying flush completed. Third, in our containerized deployment /tmp/chroma-data sat on Kubernetes ephemeral storage, so it got reclaimed — the log simply said “Persist succeeded” while the directory was already gone. Regular unit tests can never simulate these edge cases because everything runs inside a single process, and mocking silently covers up the most dangerous parts.

Designing the solution

To catch this kind of “silent data loss”, tests must run in a real multi‑process environment with network latency and disk I/O, covering the full persistence lifecycle: write → service restart → read. This is exactly where end‑to‑end testing shines.

For tooling, we ruled out pure API tests — fast but they bypass the browser and real request chain, so you can’t verify that the whole call path from a frontend trigger truly persists data to disk. We also ruled out Selenium — too heavy and its async support is poor. We settled on Playwright + pytest‑asyncio because it can:

Simulate multi‑turn conversations in a real browser
Use page.wait_for_* to control timing and capture the window where HTTP 200 is returned before data gets asynchronously lost
Directly control service process restarts during tests, verifying whether data actually landed on disk

The architecture follows “test‑as‑a‑user”: every case patterns like “open the page → send a message → close the browser → restart the service → open the page again → check if the historical memory still exists”. Only this brute‑force approach exposes persistence issues in vector databases.

Core implementation

Let’s start with a simplified app under test — a FastAPI service with a chat frontend and Chroma for memory, started with uvicorn. The code below solves one problem: how to start/stop the service with Playwright and wrap it into a reusable fixture.

# conftest.py
import subprocess
import time
import httpx
import pytest_asyncio
from playwright.async_api import async_playwright

@pytest_asyncio.fixture(scope="function")
async def app_service(tmp_path):
    # 每个测试用例拥有独立的 Chroma 持久化目录，避免数据串扰
    chroma_dir = tmp_path / "chroma_data"
    chroma_dir.mkdir()

    # 启动被测试服务，设置环境变量指定持久化路径
    proc = subprocess.Popen(
        ["uvicorn", "main:app", "--host", "127.0.0.1", "--port", "8765"],
        env={"CHROMA_PERSIST_DIR": str(chroma_dir), **__import__("os").environ},
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE
    )
    # 等待服务就绪
    async with httpx.AsyncClient() as client:
        for _ in range(30):
            try:
                resp = await client.get("http://127.0.0.1:8765/health")
                if resp.status_code == 200:
                    break
            except Exception:
                pass
            await asyncio.sleep(0.5)
        else:
            proc.kill()
            raise RuntimeError("Service did not start")

    yield {"proc": proc, "base_url": "http://127.0.0.1:8765", "chroma_dir": chroma_dir}

    # 测试结束后强杀进程，模拟生产环境可能出现的非正常退出
    proc.kill()
    proc.wait()

This fixture manages an isolated data directory per test and the process lifecycle, ensuring a clean persistence environment while simulating unexpected termination. Next is the core test case that verifies whether memory survives a restart.


python
# test_memory_persistence.py
import asyncio
import pytest
from playwright.async_api import async_playwright

pytestmark = pytest.mark.asyncio(loop_scope="module")

async def test_memory_survives_restart(app_service):
    """
    场景：用户说自己的名字，重启服务后，AI 仍然记得。
    这个测试直接揪出了 Chroma 的异步刷盘问题。
    """
    base = app_service["base_url"]

    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()

        # Step 1: 用户首次对话，留下记忆
        await page.goto(base)
        await page.fill("#user-

12 Hours Debugging Memory Drift in a LangChain+Chroma Code QA Bot

BAOFUFAN — Tue, 30 Jun 2026 12:04:48 +0000

At 2 a.m., I stared at the nonsensical reply in my terminal, a chill down my spine—just a moment ago I told it to “only look at the auth module”, and now it was recommending a frontend button color scheme. Building a local codebase Q&A bot with accurate long-conversation memory is far trickier than it seems.

Problem Deconstruction

Our scenario was clear: a Python monorepo with hundreds of thousands of lines, business logic scattered across hundreds of files. New hires and senior devs troubleshooting cross-module calls all wanted a bot that could “understand the entire codebase” and engage in multi-turn conversations—like chatting with a senior developer, not a Ctrl+F session from scratch.

The typical approach is to build a RAG pipeline with LangChain: split code files → embed → store in a vector DB → upon user query, retrieve similar chunks → stitch them into a prompt for the LLM to generate an answer. You can find tons of tutorials and have it running in 10 minutes. But in practice, as soon as the conversation exceeds three turns, the bot starts "forgetting"—mixing up previously mentioned file paths, forgetting already confirmed context, or even blending two separate questions into one answer.

The root causes are twofold:

The default ConversationBufferMemory just blindly stuffs all message history into the prompt; once it exceeds token limits, it truncates—often just the part with critical context.
The retrieval stage completely ignores conversation history, using only the latest user question to search the vector store. This renders referential queries like "modify the login function in that file" utterly useless.

The vanilla setup works for demos but collapses under real conversations. I needed a code Q&A bot that truly remembers context, plus an automated testing method to continuously verify memory accuracy—otherwise, relying on manual eyeballing after every memory logic refactor would eventually lead to failure.

Solution Design

Core stack: LangChain + Chroma + custom layered memory.

Why not Pinecone or Weaviate? Because this is a local codebase with data that must stay on-prem; and during development, we need to frequently destroy and rebuild indexes. The lightweight embedded Chroma (based on SQLite + in-memory HNSW) is fast enough, has zero deployment dependencies, and persistence is just a directory—backup is as simple as copying the folder.

For memory, I did not use LangChain's built-in SummaryMemory because summarization loses details (like an exact module path), which is fatal for code Q&A. I designed a two-layer memory structure:

Short-term window: keep the last N full conversation turns without compression to preserve referential context.
Long-term compression: for turns beyond the window, use an LLM to extract key decisions (e.g., "user is investigating concurrency issues in the payment module") and inject them as a fixed prefix into the prompt.

Most importantly, the retrieval query must be rewritten. Before hitting the vector store, I feed the user's raw question along with the last three turns into the LLM to produce a de-referenced standalone query. For example: "What exception did it throw?" becomes "What exception did the process_order function in payment/service.py throw?".

For automated testing, I designed a set of fixed conversation scripts, each ending with a question that requires memory of earlier context; then a script compares the answer with expected output to verify memory accuracy.

Core Implementation

1. Code Indexing into Chroma (persisted locally)

This code handles converting code files into searchable vectors; the key is the splitting strategy—using Language.PYTHON's separators to ensure chunks fall on function/class boundaries.

from langchain.document_loaders import DirectoryLoader, TextLoader
from langchain.text_splitter import Language, RecursiveCharacterTextSplitter
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import Chroma
import os

# 加载代码目录，排除测试和虚拟环境
loader = DirectoryLoader(
    "./src",
    glob="**/*.py",
    loader_cls=TextLoader,
    loader_kwargs={"encoding": "utf-8"},
    exclude=["**/test_*", "**/__pycache__/*"],
)
docs = loader.load()

# 按 Python 语法边界切分，chunk 大小 1000，重叠 200 字符保留上下文
splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON,
    chunk_size=1000,
    chunk_overlap=200,
)
chunks = splitter.split_documents(docs)

# 使用本地 embedding 模型，避免网络依赖
embedding_model = HuggingFaceEmbeddings(
    model_name="BAAI/bge-base-zh-v1.5",
    model_kwargs={"device": "cpu"},
    encode_kwargs={"normalize_embeddings": True},
)

# 存入 Chroma，持久化目录指定即可重启保留
vectordb = Chroma.from_documents(
    documents=chunks,
    embedding=embedding_model,
    persist_directory="./chroma_code_index",
    collection_name="code_base",
)
vectordb.persist()

2. Q&A Chain with Dual-Layer Memory

This code tackles "how to make the LLM remember history without blowing the token limit" by combining a short-term window with long-term compression, and injecting a retrieval rewriting step.

from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferWindowMemory, ConversationSummaryMemory
from langchain.chat_models import ChatOpenAI
from langchain.prompts import PromptTemplate

llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)

# 短期记忆：保留最近 4 轮完整对话
short_memory = ConversationBufferWindowMemory(
    k=4,
    memor

Redis Persistence Pitfalls: A Data Loss Bug That Took Me 6 Hours to Uncover

BAOFUFAN — Tue, 30 Jun 2026 01:03:56 +0000

At 2 a.m., the on-call phone jolted me awake. The SRE from the user service sounded panicked: “Redis was restarted on production, and now a ton of user sessions are gone — everyone’s logged out, the customer support lines are exploding.” As I pulled up the monitoring dashboards, my heart sank. This Redis instance was dedicated to storing user sessions. A year ago, we had explicitly requested persistence to be enabled, and it was configured with both RDB and AOF for safety — so how on earth did we lose data? By daybreak, I finally pieced together the root cause. Along the way, I built an automated verification system with Pytest and Docker to ensure this never happens again. Let me walk you through what went wrong and how this validation tool can help you avoid the same trap.

Breaking Down the Problem: Why Does Data Still Disappear When Persistence Is Enabled?

The setup was straightforward: we used Redis to store web sessions and required data to survive a restart. The ops team had configured RDB (save 900 1) and AOF (appendfsync everysec), which looked rock solid. But the failure sequence went like this:

The container had been running for 2 hours; the last RDB save was 30 minutes ago.
A rolling restart was triggered; the Redis process was gracefully shut down with docker stop.
After the restart, we discovered that among the nearly 20,000 sessions written in the past 30 minutes, around 10,000 were lost.

The root cause is that the combination of RDB and AOF does not handle the scenario of “writing a large volume of data in a short time and then immediately restarting.” The save 900 1 rule means that if only 1 key changes, Redis would wait 900 seconds before saving to disk. Meanwhile, AOF with appendfsync everysec flushes every second, but docker stop gives the process 10 seconds to exit by default. Upon receiving SIGTERM, the Redis process tries to flush its AOF buffer to disk. The problem is: if the amount of data is too large and the flush takes longer than 10 seconds, Docker sends SIGKILL, leaving an incomplete AOF file. When Redis restarts and detects the corrupted AOF file, it either truncates the tail or skips repair based on its configuration — thus the unflushed data simply vanishes.

Why don’t typical approaches work? In the past, to test persistence, we would spin up a local Docker container, SET a few keys, restart, and see if the data was still there. This “eyeball test” cannot simulate massive writes and race conditions near timeout boundaries, nor can it cover various “unexpected kill” scenarios. So we absolutely need an automated, repeatable validation method that covers real-world failure modes.

Designing the Solution: Parameterized Validation with Pytest and Docker

I needed to do three things:

Quickly spin up a Redis container with a predefined persistence policy.
Simulate real write pressure.
Kill the container in different ways, then restart it and check data consistency.

For the test framework, I went with Pytest because of its powerful parameterization — you can easily generate test combinations of different persistence configurations and failure modes. For container management, I chose testcontainers-python. It automatically handles container lifecycles, leaves no dirty environments, and is easy to run in CI. Why not use docker-compose? Because I needed to run dozens of configurations with the same image; writing a bunch of YAML files would be tedious and inflexible. With testcontainers, I can dynamically create configurations in code and even mount a custom redis.conf.

The overall architecture: a conftest.py provides a parameterizable Redis container fixture. Test cases verify two scenarios: “graceful shutdown” and “unexpected power loss.” Finally, using Pytest’s --csv or Allure, we generate a report that clearly shows data loss under each strategy.

Core Implementation: Building the Automated Validation Step by Step

Step 1: Define a container fixture that can configure Redis on demand

This snippet solves the problem of “same image, different persistence policies.” We use testcontainers’ Redis module but directly pass command-line arguments or mount a custom configuration file. For precise control, I chose to mount a custom redis.conf.

# conftest.py
import pytest
from testcontainers.redis import RedisContainer
from testcontainers.core.container import DockerContainer
import tempfile
import os

def _write_conf(conf_content: str) -> str:
    """把配置写入临时文件，返回宿主机路径"""
    tmp = tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.conf')
    tmp.write(conf_content)
    tmp.close()
    return tmp.name

@pytest.fixture
def redis_container(request):
    """
    按参数化需求创建Redis容器，挂载自定义redis.conf
    request.param 格式示例: {"save": "1 1", "appendfsync": "no", "aof": "no"}
    """
    params = getattr(request, 'param', {})
    save_rule = params.get('save', '')          # 如 "900 1"
    appendfsync = params.get('appendfsync', 'no')
    use_aof = params.get('aof', 'no')

    conf_lines = []
    if save_rule:
        conf_lines.append(f"save {save_rule}")
    else:
        conf_lines.append("save \"\"")           # 完全关闭RDB
    if use_aof.lower() == 'yes':
        conf_lines.append("appendonly yes")
        conf_lines.append(f"appendfsync {appendfsync}")
    else:
        conf_lines.append("appendonly no")

    conf_path = _write_conf("\n".join(conf_lines))

    # 构建容器，挂载配置并启动
    container = DockerContainer("redis:7-alpine")
    container.with_volume_mapping(conf_path, "/usr/local/etc/redis/redis.conf")
    container.with_command("redis-server /usr/local/etc/redis/redis.conf")
    container.with_exposed_ports(6379)
    container.start()

    redis_host = container.get_contain

LangChain Chatbot Session Mix-Up: When 100 Concurrent Users Got Each Other's Replies

BAOFUFAN — Mon, 29 Jun 2026 12:04:32 +0000

At 2 AM, my phone blared. It was our ops guy, pulling me out of a dream: “Users are posting screenshots in the support group — they’re seeing other people’s order numbers inside our AI assistant. The group is going crazy.”

I logged onto the server, checked the logs, and my brain went blank. Multiple requests had their conversation histories all jumbled together. User A asked “Where is my delivery?” and the bot answered with User B’s address. It felt like locking your door, coming home, and finding a stranger sitting on your couch.

Breaking It Down: Why User Sessions “Cross Wires”

Our setup was straightforward: a multi-turn chatbot built with LangChain, backed by a large language model, served via FastAPI. Each user was supposed to have an independent conversation memory, never interfering with one another. Unit tests and feature tests all passed. Management gave the green light, and we rolled out a canary release.

The moment traffic hit, everything fell apart.

The root cause lies in the most common usage of LangChain’s ConversationBufferMemory: it stores conversation history in Python’s in-process memory. When concurrent requests pour in, if you don’t deliberately isolate by session_id, all requests share the same memory instance. User A’s history gets contaminated with User B’s messages, the model receives dirty context, and data cross-contamination occurs.

There’s an even sneakier trap: even if you deepcopy the memory on every request, if you fail to atomically combine “load history by session” and “write back after the call,” concurrent writes will still overwrite each other. User B can end up reading leftover history from User A. This problem never shows up in sequential single-user testing; it waits patiently until concurrency arrives.

Design Decisions: Why We Didn’t Pick the Alternatives

What we fundamentally needed was a conversation-history store keyed by session_id, persistent across requests, and safe for concurrent reads and writes.

A few paths were on the table:

In-process dict + lock: the simplest approach, but it can’t span processes, all data disappears on restart, and the lock would strangle concurrency performance. Immediately ruled out.
PostgreSQL / MySQL: they can store the data, but chat scenarios are read-heavy with occasional writes. Frequent SELECTs to retrieve history would amplify database pressure, and adding JSON serialization on top makes the latency not worth it.
Redis: in-memory read/write speed, native support for List data structures to hold multiple messages, the ability to set TTL per session for automatic expiration, and a single instance can easily handle tens of thousands of concurrent operations. On top of that, LangChain’s BaseChatMessageHistory interface already provides a slot to swap in alternative storage, making the implementation cost extremely low.

So our final choice: implement RedisChatMessageHistory, store each session’s messages in a Redis List keyed as messages:{session_id}, and then let LangChain’s RunnableWithMessageHistory automatically complete the cycle of “load on request, write after response.”

Core Implementation: Three Steps to Isolation

Step 1: Implement a custom RedisChatMessageHistory

This piece solves how to persist conversation history into Redis. We inherit from BaseChatMessageHistory, use a Redis List to hold the messages, and serialize each message as a JSON string.

import json
import logging
from typing import List

import redis
from langchain_core.chat_history import BaseChatMessageHistory
from langchain_core.messages import BaseMessage, messages_from_dict, message_to_dict

logger = logging.getLogger(__name__)


class RedisChatMessageHistory(BaseChatMessageHistory):
    """使用 Redis List 存储对话历史，每个 session_id 一个 key"""

    def __init__(
        self,
        session_id: str,
        redis_client: redis.Redis,
        ttl: int = 3600,          # 默认 1 小时过期，避免僵尸 key
        key_prefix: str = "chat_history:",
    ):
        self.session_id = session_id
        self.redis_client = redis_client
        self.ttl = ttl
        self.key = f"{key_prefix}{session_id}"

    @property
    def messages(self) -> List[BaseMessage]:
        # 从 Redis List 中读取全部消息，范围 0 到 -1 保证不丢数据
        raw_messages = self.redis_client.lrange(self.key, 0, -1)
        # 每一条存储时是 JSON 字符串，这里反序列化回 LangChain 消息对象
        dict_messages = [json.loads(m) for m in raw_messages]
        return messages_from_dict(dict_messages)

    def add_message(self, message: BaseMessage) -> None:
        # message_to_dict 将 BaseMessage 转成 dict，再 dump 成 JSON 字符串
        serialized = json.dumps(message_to_dict(message))
        # 写入 Redis List 右侧，保持时间顺序
        self.redis_client.rpush(self.key, serialized)
        # 刷新过期时间，活跃用户的历史保持不过期
        self.redis_client.expire(self.key, self.ttl)

    def clear(self) -> None:
        self.redis_client.delete(self.key)

Step 2: Create a history factory to load and write per session automatically

This part solves how each request gets its own history instance. We need a factory function so that RunnableWithMessageHistory instantiates our RedisChatMessageHistory using the session_id on every call.

from langchain_core.runnables.history impo

Redis Crash Recovery Testing: The 2 Bugs Manual Testing Missed

BAOFUFAN — Mon, 29 Jun 2026 01:04:28 +0000

At 3 a.m., an ops colleague called urgently: "The AI customer service bot's memories are all messed up—users were just chatting about their orders, and the next second the bot forgot everything. We found that Redis had crashed and restarted." I opened monitoring and saw that after Redis restarted, many session memories had been partially lost, but not completely—it was like "selective forgetting." This was worse than total loss—no one noticed the data was corrupted until users started complaining. Ultimately, the root cause pointed to one thing: we had never truly automated the verification of memory consistency after Redis crash recovery. Every time, we manually typed redis-cli to test a few keys and concluded "probably okay."

The deception of manual testing is that the failure scenarios you simulate are always idealized. You flush, kill, check a few data entries, and everything looks fine. But in real production, Redis can lose partial data in ways you never imagined—due to differences between RDB and AOF, fsync policies, and the loading order of mixed persistence during restarts. In this article, I'll break down the painful lesson and the solution: Redis crash recovery consistency verification based on Pytest + Testcontainers, and reveal the two hidden bugs that could only be caught by automated testing.

Problem Breakdown — Why Manual Testing Misses "Selective Forgetting"?

We use Redis as the short-term memory store for LLM. Each session's memories are stored in a List under the key session:<id>:messages, and each message is a serialized JSON. The recovery requirement is simple:

No matter how Redis crashes and restarts, the memories must either be all there or all gone (when full persistence is used), never half-present.

The real scenario is more complex: the failure could happen right after an RDB snapshot is completed, while AOF is rewriting, or in the middle of mixed persistence. When simulating manually, it's nearly impossible to kill Redis at exactly those precise moments. You can only do a few docker restarts, then LRANGE to check, see the data still there, and sign off. But the key to reproducing "selective forgetting" lies in:

Bug1: With only RDB enabled (no AOF), after a restart, all incremental writes since the last snapshot are lost, while developers assumed automatic saving.
Bug2: With AOF enabled but appendfsync everysec, when the process is killed abruptly, the last 1–2 seconds of write operations disappear, causing the memory list to lack a few tail entries, but the old data can still be read from the list, making it very difficult to spot with the naked eye.

These two issues are almost invisible with manual redis-cli get because you tend to pick a specific key to check, and if that key happens to fall within the snapshot range or before the AOF fsync, you'll be tricked. Only automated, parameterized, and repeatable fault injection testing can systematically expose them.

Solution Design — Why Pytest + Testcontainers?

To simulate real Redis crash recovery, there are several hard requirements:

Each test must start with a clean Redis instance with configurable persistence policies.
Be able to force-kill the Redis process at any point in the code execution, then restart and verify data integrity.
Tests need to be fully programmable and CI-integrable, not manual commands.

Common alternatives were all rejected:

fakeredis: purely in-memory simulation, no real persistence or process crashes—totally incapable of testing fault recovery.
docker-compose + manual scripts: decent environment isolation, but fault injection is limited to docker kill, can't trigger at precise code logic points, and hard to parameterize and assert.
Direct host Redis operations: dirty environment, interference between tests, impossible for CI.

Testcontainers for Python hits all these points perfectly: it starts a real Redis container directly in test code, we can obtain a connection via with redis_container, and at any moment call container.stop() then container.start() to simulate a crash restart. Combined with Pytest's fixture mechanism, each test case gets an independent Redis lifecycle, and the persistence configuration file can be injected via mounted volumes. The entire verification becomes repeatable unit tests, with fault injection granularity down to the line of code.

Core Implementation — Building Memory Consistency Tests Step by Step

Let's dive into the code. Assume a memory storage class RedisMemoryStore provides append_message(session_id, msg) and get_messages(session_id), backed by a Redis List. Now we need to write a test to verify its flaws under the "RDB without AOF" configuration.

Dependency Installation and Container Fixture (conftest.py)

This fixture solves the “fresh Redis per test” problem and allows passing persistence configuration via an environment variable.

# conftest.py
import pytest
import redis
from testcontainers.redis import RedisContainer

# 自定义 Redis 容器，允许传入配置文件
class ConfigurableRedisContainer(RedisContainer):
    def __init__(self, image="redis:7-alpine", port=6379, config_path=None):
        super().__init__(image=image, port=port)
        self.config_path = config_path

    def _configure(self):
        super()._configure()
        if self.config_path:
            # 挂载配置文件到容器
            self.volumes = {self.config_path: {"bind": "/usr/local/etc/redis/redis.conf", "mode": "ro"}}
            self.with_command("redis-server /usr/local/etc/redis/redis.conf")

@pytest.fixture(scope="function")
def redis_no_aof():
    """仅开启 RDB 的 Redis 实例（Bug1 专用）"""
    # 配置文件内容：只打开 RDB，关闭 AOF
    config = """
    save 900 1
    save 300 10
    save 60 10000
    appendonly no
    """
    import tempfile, os
    with tempfile.NamedTemporaryFile(mode='w', suffix='.conf', delete=False) as f:
        f.write(config)
        config_path = f.name
    try:
        container = ConfigurableRedi

After 3 Years of Mocking Redis, We Missed 60% of Edge Cases in Memory Store Tests

BAOFUFAN — Sun, 28 Jun 2026 12:03:44 +0000

2 AM. Production alarms start screaming — the chatbot's memory module is dropping user contexts. Digging into the logs: JSONDecodeError, caused by inconsistent datetime serialization in a session object. “But the tests are all green!” I said. A teammate replied, “Our Redis tests are always mocked.” That sent a chill down my spine.

Mocking shields you from every real‑world failure, while CI keeps handing you false green lights.

Why Mocked Tests Make Your Redis Memory Storage a Paper Tiger

LLM memory storage almost always uses Redis: low latency, built‑in expiration, atomic operations. The typical pattern is giving each session a key, serializing the conversation history to JSON, stuffing it in, and setting a TTL.

A mocked test looks like this: use mock.patch to swap out redis.Redis, then verify that set/get were called with the right arguments. If all you need is “did I invoke Redis?”, that’s enough. But the real‑world traps that mocks can’t replicate are exactly where things break:

Serialization / deserialization edge cases — datetime, Decimal, custom objects … different Redis client versions behave differently. Mock won’t exercise those.
Connection timeouts and retries — setex times out, raises TimeoutError. Does your retry logic actually work? Mock just returns True; it will never hit the ConnectionError branch.
Memory eviction and expiry policies — when maxmemory-policy is allkeys-lru, a GET can suddenly return None. A mock dictionary keeps everything forever.
Concurrency and atomicity — setnx for distributed locks. Two coroutines racing to acquire the lock? A mock dict is thread‑safe; real Redis exposes the race condition.

Over the last six months we hit three memory‑storage incidents in production: a JSON deserialization crash, a TTL accidentally set to 0 that made keys live forever, and an unhandled MOVED redirection during a Redis failover. All three times, the mocked tests passed.

That’s why I say we missed 60% of edge cases — no exaggeration.

Why Testcontainers Instead of In‑Memory Redis or Shared Environments

If you want to test real Redis behavior, you have a few options:

Shared integration environment — multiple branches share one Redis instance, data gets mixed up, you wait for ops to reset between runs. Slow and unstable.
Embedded Redis (like embedded-redis or fakeredis) — fakeredis is excellent for pure‑Python simulation, but it’s still a simulation. Advanced features like eviction or failover don’t match real Redis 100%. If you’ve been burned before, you know fakeredis's expire behavior can diverge from real Redis.
Testcontainers — spin up a real Redis container per test session, tear it down when done. ✅ Real behavior ✅ Total isolation ✅ Near‑instant startup after the initial docker pull.

Testcontainers standardizes “pull Docker images in tests.” Our project landed on Testcontainers for Python because it integrates with pytest beautifully, and it works identically locally and in CI (as long as Docker is available).

Core Implementation: From Spinning Up a Container to Deterministic Integration Tests

You only need pip install testcontainers[redis] redis, then follow along.

1. A Reusable Redis Container Fixture

What this solves: share one Redis container across a test class, but give each test case a separate prefix or database to avoid data cross‑contamination. Using scope="class" balances startup cost and isolation.

# tests/conftest.py
import pytest
from testcontainers.redis import RedisContainer
from redis import Redis, ConnectionPool
import uuid

@pytest.fixture(scope="class")
def redis_container():
    """启动 Redis 7 容器，暴露随机端口，自动回收"""
    with RedisContainer("redis:7-alpine") as container:
        # 获取映射后的端口和主机
        host = container.get_container_host_ip()
        port = container.get_exposed_port(6379)
        yield host, port

@pytest.fixture
def redis_client(redis_container):
    """每个测试函数一个独立的 Redis 客户端，自动选择未使用的 db 编号"""
    host, port = redis_container
    # db 编号隔离：取 hash 保证不冲突，实际项目可用 0-15 轮转
    test_db = abs(hash(str(uuid.uuid4()))) % 16
    pool = ConnectionPool(host=host, port=port, db=test_db, decode_responses=True)
    client = Redis(connection_pool=pool)
    yield client
    # 清理当前 db，不留痕
    client.flushdb()
    pool.disconnect()

Why RedisContainer("redis:7-alpine")? Alpine images are small and pull quickly; pinning to major version 7 prevents test behavior from shifting with a new Redis release. If your CI environment already caches the image, cold start is under 3 seconds.

2. A Real, Disk‑Backed Memory Storage Class

What this solves: a MemoryStore that will actually be used in production, containing serialization, expiration, timeout retries — things that mocking never tests. This class is a simplified version of our real business code.

# app/memory.py
import json
from datetime import datetime
from typing import Any, Optional
from redis import Redis
from redis.exceptions import TimeoutError, ConnectionError

class MemoryStore:
    def __init__(self, client: Redis, default_ttl: int = 3600):
        self.client = client
        self.default_ttl = default_ttl

    def save(self, session_id: str, context: dict) -> bool:
        """Save session context, serialize datetime to ISO format."""
        try:
            serialized = json.dumps(context, default=self._serializer)
            return self.client.setex(session_id, self.default_ttl, serialized)
        except (TimeoutError, ConnectionError):
            # Retry logic
            return False

    def load(self, session_id: str) -> Optional[dict]:
        """Load and deserialize session context."""
        try:
            data = self.client.get(session_id)
            if data is None:
                return None
            return json.loads(data, object_hook=self._deserializer)
        except (TimeoutError, json.JSONDecodeError):
            return None

    @staticmethod
    def _serializer(obj):
        if isinstance(obj, datetime):
            return obj.isoformat()
        raise TypeError(f"Type {type(obj)} not serializable")

    @staticmethod
    def _deserializer(dct):
        # Rebuild datetime from ISO strings
        for key, value in dct.items():
            if isinstance(value, str):
                try:
                    dct[key] = datetime.fromisoformat(value)
                except ValueError:
                    pass
        return dct

3. Integration Tests That Cover the 60% Hidden Edge Cases

What this set of tests proves: serialization exceptions, timeout retries, key absence after eviction, and TTL verification — real behaviors that mock tests would completely miss.

# tests/test_memory_integration.py
import pytest
import time
from app.memory import MemoryStore
from redis.exceptions import TimeoutError, ConnectionError

class TestMemoryStoreIntegration:
    def test_save_and_load_basic(self, redis_client):
        store = MemoryStore(redis_client)
        session_id = "user:123"
        context = {"name": "Alice", "timestamp": datetime.now()}
        assert store.save(session_id, context)
        loaded = store.load(session_id)
        assert loaded["name"] == "Alice"
        # 确保 datetime 正确反序列化
        assert isinstance(loaded["timestamp"], datetime)

    def test_json_decode_error_caught(self, redis_client):
        store = MemoryStore(redis_client)
        session_id = "corrupted:session"
        # 直接写入非法 JSON，模拟生产环境脏数据
        redis_client.set(session_id, "{this is not json")
        result = store.load(session_id)
        assert result is None  # 应捕获异常并优雅降级

    def test_ttl_expiry(self, redis_client):
        store = MemoryStore(redis_client, default_ttl=1)  # 1 秒过期
        session_id = "ephemeral:user"
        store.save(session_id, {"data": "tmp"})
        time.sleep(1.1)
        assert store.load(session_id) is None  # Key 应已过期

    def test_connection_timeout_retry(self, redis_client, mocker):
        """模拟 TimeoutError 时重试逻辑被触发"""
        store = MemoryStore(redis_client)
        # 第一次调用抛出 TimeoutError，第二次正常
        mocker.patch.object(redis_client, 'setex', side_effect=[TimeoutError(), True])
        result = store.save("retry:session", {"key": "val"})
        assert result is False  # 当前实现：超时直接返回 False
        # 若想验证重试次数，需增强实现，但至少覆盖了异常分支

    @pytest.mark.parametrize("policy", ["allkeys-lru", "volatile-lru"])
    def test_eviction_returns_none(self, redis_container, policy):
        """验证内存淘汰后 GET 返回 None（需要真实 Redis）"""
        host, port = redis_container
        # 为这个测试启动一个内存极小的容器
        # 实际项目中用 Testcontainers 再起一个定制 Redis，这里省略详细配置
        # 关键点：当 maxmemory 写满时，前一个 key 可能被逐出
        # (示例演示思路，完整实现需定制 Redis 配置)
        pass  # 占位表示真实测试结构

Notice how we test the actual serialization format (datetime → ISO → back), graceful degradation on corrupt data, real TTL expiration, and timeout branches. These are exactly the paths that mock‑based tests ignore.

The “No Excuses” CI Strategy: Speed Meets Reality

“But Testcontainers is slow.” That’s only true if you start a container for every single test. With a class‑scoped fixture, you pay the container startup cost once per test class (usually under 3 seconds), and each test merely switches databases. In CI we run these integration tests alongside unit tests; the total suite still completes in under 2 minutes.

We configure CI with a step that caches the Docker image:

# .github/workflows/test.yml (fragment)
- name: Cache Redis image
  uses: actions/cache@v3
  with:
    path: ~/.docker/images
    key: redis7-alpine-${{ hashFiles('**/requirements.txt') }}
- name: Pull image (if not cached)
  run: docker pull redis:7-alpine
- name: Run tests
  run: pytest --cov=app tests/

Even without caching, the pull is a one‑time cost. The real payoff is catching memory‑store bugs before they wake you up at 2 AM.

Closing: Replace the Fake Green Light with Real Confidence

We’ve been using this Testcontainers‑backed approach for six months now. The three production incidents we had? All of them would have been caught by the integration tests above. No more phantom passes. No more wondering why production “suddenly” broke while CI was perfectly green.

If your team is still mocking Redis for memory storage, do this one thing: pull in Testcontainers, write a single integration test that saves a datetime and reads it back. In five minutes you’ll see the difference between “the mock says it worked” and “it actually works.”

Sleep better. Ship with confidence. And let your tests actually talk to the same Redis that runs in production.

How I Fixed LangChain’s Agent Memory Loss with 23 Pytest Cases

BAOFUFAN — Sun, 28 Jun 2026 01:03:24 +0000

At 1 AM, my boss @-ed me in the company chat: “What’s wrong with the customer service bot? The user just said their name, and the very next message it asks again — like a goldfish.” I stumbled to my laptop and pulled up the logs. Sure enough, the context queue for that session_id in Redis was clean as a whistle, as if nothing had ever been written. That was the fourth “memory evaporation” incident this month. Every previous time, we’d papered over it with a service restart and a redeploy. But that night I decided to stop coddling the bug — I was going to test the AI agent’s context persistence logic from top to bottom with Pytest.

Where exactly does an agent’s memory get lost?

An AI agent’s “memory” is basically a list of messages. With each turn, you push the user’s input and the AI’s reply into that list. On the next request, you pull out the history and prepend it to the prompt so the model knows what was said before. A typical implementation looks like this:

from langchain.memory import ConversationBufferMemory
memory = ConversationBufferMemory(return_messages=True)
# After each turn: memory.save_context({"input": user_msg}, {"output": ai_msg})
# Later: memory.load_memory_variables({}) to get the history

When we wire this up with Redis for persistence (e.g. using RedisChatMessageHistory), memory should survive as long as Redis does. But in the real world, sometimes a conversation gets routed to a different machine, and fields get lost in the serialized JSON or Pickle payload. Sometimes the async connection pool gets exhausted and save_context returns without really flushing. Even spookier: when return_messages=True, load_memory_variables can return SystemMessage and HumanMessage objects, but if you serialize them with pickle, the slightest difference in class definitions on the deployment server causes a silent deserialization failure — you end up with an empty list and not a single error.

These problems share one trait: they only surface under multi-turn, concurrent conversations, and manual testing can never reproduce them. Debug locally step by step and everything works fine. Deploy to production and you get intermittent amnesia. Without an automated test suite that simulates real, multi-turn interactions, you’re effectively planting a time bomb in every release.

Design decisions: Why Pytest + fakeredis instead of a real Redis cluster

I needed a test setup that could run in CI, execute in milliseconds, and reliably reproduce every edge case. The options on the table:

Real Redis + Testcontainers — Spins up a legitimate Redis container, very close to production. But each test requires a few seconds to start the container, which slows down CI pipelines noticeably. It also ties you to Docker; some teammates use M1 Macs, and certain images cause compatibility headaches.
unittest + mock — Swap out the Redis client with Python’s standard mock library. It can test the logic, but the code becomes verbose (mock every single Redis command), and mocks don’t reproduce the real behavior of serialization/deserialization.
Pytest + fakeredis — fakeredis is a pure-Python Redis simulator that supports over 90% of common commands, fully emulating serialization, expiration, and data structures — with zero external dependencies. Combined with Pytest’s fixtures, parametrization, and pytest-asyncio, you can polish every corner of the memory storage logic.

Another reason not to use a real Redis cluster: some of the scenarios I wanted to test depended on specific Redis error responses (e.g. OOM, connection timeouts). With fakeredis, simulating those anomalies is actually easier than with a real instance, because you can inject specific faults. This way the test environment becomes “a version of production that knows your code better than you do” — exposing those sporadic boundary conditions.

The final architecture is simple: a MemoryStore abstract class that encapsulates the agent’s memory operations (add messages, get history, trim, clear). The underlying implementation can be Redis or an in-memory dict. During testing, we inject fakeredis; in development, we reuse the exact same test suite to validate any new storage backend.

Core implementation: from fixtures to multi-turn conversation tests

Setting up an isolated memory store so every test starts fresh

We start with a Pytest fixture that provides an isolated, async memory store, guaranteeing no cross-test interference:

# conftest.py
import pytest
import fakeredis.aioredis  # async fakeredis
from myapp.memory_store import RedisMemoryStore

@pytest.fixture
async def memory_store():
    fake_redis = fakeredis.aioredis.FakeRedis(decode_responses=True)
    store = RedisMemoryStore(redis_client=fake_redis)
    yield store
    await store.clear_all()  # clean up after each test
    await fake_redis.aclose()

We built RedisMemoryStore ourselves, avoiding a direct dependency on LangChain’s History classes. This lets us test serialization logic in isolation and makes it easy to swap out for DynamoDB or PostgreSQL later.

Verifying the most critical behavior: you can save and retrieve what you stored

# test_memory_basic.py
import pytest
from myapp.schemas import ChatMessage

pytestmark = pytest.mark.asyncio  # the whole module is async

async def test_save_and_retrieve_messages(memory_store):
    session_id = "session-001"
    messages = [
        ChatMessage(role="user", content="My name is Wang"),
        ChatMessage(role="assistant", content="Got it, Wang"),
    ]
    await memory_store.add_messages(session_id, messages)

    history = await memory_store.get_history(session_id)

    assert len(history) == 2
    assert history[0].role == "user"
    assert history[0].content == "My name is Wang"
    assert history[1].role == "assistant"

Here ChatMessage is a Pydantic model. Serialization uses .json() and stores the result in a Redis Hash, avoiding Pickle and its class-versioning issues altogether.

Simulating real-world multi-turn conversations concurrently

Now the part that catches the “goldfish” bug: multiple interleaved sessions, each with multiple turns, running under asyncio.gather to trigger race conditions.

async def test_concurrent_multi_session_isolation(memory_store):
    session_a = "sess-A"
    session_b = "sess-B"

    await memory_store.add_messages(session_a, [ChatMessage(role="user", content="From A1")])
    await memory_store.add_messages(session_b, [ChatMessage(role="user", content="From B1")])

    # Simulate two sessions continuing at the same time
    async def continue_session(sid, content):
        await memory_store.add_messages(sid, [ChatMessage(role="assistant", content=content)])

    await asyncio.gather(
        continue_session(session_a, "Reply A"),
        continue_session(session_b, "Reply B"),
    )

    hist_a = await memory_store.get_history(session_a)
    hist_b = await memory_store.get_history(session_b)

    assert len(hist_a) == 2
    assert all(h.role in ("user", "assistant") for h in hist_a)
    assert hist_a[-1].content == "Reply A"

    assert len(hist_b) == 2
    assert hist_b[-1].content == "Reply B"

Testing what happens when Redis misbehaves

With fakeredis we can also simulate infrastructure failures:

async def test_graceful_fallback_on_redis_timeout(memory_store):
    # Forge the fake client to raise on the next write
    memory_store.redis_client.set = AsyncMock(side_effect=ConnectionError("timeout"))

    with pytest.raises(MemoryStoreBackendError):
        await memory_store.add_messages("s1", [ChatMessage(role="user", content="this will fail")])

In total, I wrote 23 such cases — covering session isolation, message trimming when a conversation grows too long, clearing history, TTL expiration, serialization edge cases, and concurrent write collisions. Every time a new “memory evaporation” pattern appeared, I added a test that reproduced it, fixed the code, and watched it go green.

Results and takeaways

After integrating the test suite into our CI pipeline, memory-related incidents disappeared from the #oncall channel. The boss now knows me as “the guy who ended the goldfish era.” More importantly, the team now treats memory persistence as a first-class concern — every new storage adapter gets validated against the same 23 cases before it ever touches production.

If your LangChain agent ever “forgets” what the user just said, don’t reach for a redeploy. Write a test that replays the exact conversation pattern, break it, and only then fix it. You’ll sleep better — and your users won’t have to introduce themselves three times.

From 2 Hours to 10 Minutes: Pytest Parametrization for LLM Memory Tests

BAOFUFAN — Sat, 27 Jun 2026 12:04:21 +0000

It was 1 a.m. The product manager was furiously @mentioning me in the group chat: “A user reported that the agent got their dietary preferences wrong – they clearly said no cilantro, but every recommended dish is packed with cilantro.” Squinting at the logs, I discovered that a critical record was missing from the memory recall. To reproduce the issue, I spent half an hour crafting curl commands, manually scrolling through the returned memory list line by line in the terminal. That’s when I realized – if I kept testing like this manually, a disaster was inevitable.

Breaking Down the Problem

Anyone building LLM agents knows that memory storage is the key to personality consistency. Whether you use vector databases (Chroma, Pinecone) for long-term memory or LangChain’s ConversationBufferMemory for session windows, every time you tweak the retrieval strategy – changing the embedding model, adjusting top_k, shifting the similarity threshold, adding summarization compression – you must re-verify two core metrics:

Recall: Are all the memories that should be remembered actually recalled, with no missing ones?
Precision: Is the retrieved content truly relevant and free of hallucinations?

If you’ve ever done manual testing, you know the pain: you first inject conversations one by one into the memory store according to a test case table, then manually ask questions, and finally scrutinize the returned content to check if it contains the expected key information. Five scenarios take over an hour; a full regression before a release easily takes two hours. The worst part: human eyes get tired and miss things. “But I thought I tested that” becomes the most common excuse when something breaks.

Why Conventional Approaches Fail

Why don’t conventional approaches work? You might suggest writing a few hard-coded cases with unittest. But LLM memory test scenarios often involve combinatorial explosions – varying conversation history lengths, semantically similar distractors, different recall limits. unittest’s parameterization capabilities are too weak; maintaining fixtures and a pile of test data makes you want to delete the repository. Running a quick check in a Jupyter Notebook lacks regression capability – you run it and throw it away. What we need is a repeatable, extensible, and clear-reporting automated validation system.

Solution Design

The tech choice is straightforward: Pytest + a pluggable memory abstraction.

Pytest: Its parametrization (@pytest.mark.parametrize) is a natural fit for generating batches of test cases combining “injected conversation × query condition × expected result”. Fixtures elegantly manage the lifecycle of memory instances. The plugin ecosystem is rich – generating HTML reports or integrating with CI costs nothing.
Why not unittest? Its ddt and parameterized are like stepchildren – you end up with a pile of decorators, and data-driven testing is so painful that newcomers want to give up right away.
Why not just use online monitoring like LangSmith? That’s for production observability. What we need is offline regression: after changing a single line of retrieval code, we can run the entire test suite in under 5 minutes, leaving no hidden risks for production.

Architecture idea: extract an abstract memory interface to avoid coupling to a real vector database, so unit tests don’t depend on external services and execute in seconds. Then build an injectable test dataset that defines “preloaded conversation list”, “query string”, “set of expected memory_id to recall”, and “expected key text snippets”. Pytest takes this data and automatically runs the flow: “load memories → trigger retrieval → assert both metrics”. This pattern can easily be adapted to any memory backend.

Core Implementation

Let’s dive into runnable code, broken down by module, with each snippet explaining what problem it solves.

1. Abstract a minimal memory store, removing external dependencies

To make tests run instantly on any dev machine, we first write a minimal FakeVectorMemory that stores memories in a list and simulates “semantic retrieval” via simple keyword matching. In a real project, you would just replace retrieve with a call to vectorstore.similarity_search().

# fake_memory.py —— 解决“如何离线模拟向量检索”的问题
from typing import List, Dict, Optional

class FakeVectorMemory:
    """模拟向量记忆存储：插入带ID的记忆，检索返回包含query关键词的记录"""

    def __init__(self):
        self._memories: List[Dict] = []  # 每条记忆：{"memory_id": str, "content": str}

    def add_memory(self, memory_id: str, content: str) -> None:
        """插入一条记忆"""
        # 过滤重复 ID，防止误报召回率
        if not any(m["memory_id"] == memory_id for m in self._memories):
            self._memories.append({"memory_id": memory_id, "content": content})

    def retrieve(self, query: str, top_k: int = 5) -> List[Dict]:
        """
        模拟检索：找出 content 包含 query 中任一非停用词的记忆。
        真实场景这里会调用 embedding + 向量搜索。
        """
        # 简易分词（生产环境请用真正的 tokenizer）
        keywords = [w.strip().lower() for w in query.split() if len(w.strip()) > 1]
        if not keywords:
            return []

        matched = []
        for mem in self._memories:
            content_lower = mem["content"].lower()
            # 只要内容包含任意关键词就认为“命中”，模拟相似度匹配
            if any(kw in content_lower for kw in keywords):
                matched.append(mem)
                if len(matched) >= top_k:
                    break
        return matched

2. Manage memory lifecycle with a Pytest fixture

Before each test case, the reset_memory fixture provides a fresh, empty memory instance, preventing data contamination between tests. This is where fixtures are more elegant than setUp/tearDown.


python
# conftest.py —— 解决“每个测试用例都需要干净记忆环境”的问题
import pytest
from fake_memory import

Caging Redis Cache Inconsistency: A 6-Hour Debugging Nightmare Solved with pytest and In-Memory Snapshots

BAOFUFAN — Sat, 27 Jun 2026 01:03:35 +0000

At 2:17 AM, my phone buzzed so violently it rattled the whole desk. Users were cursing in the support tickets — they’d update their nickname, hit refresh a few minutes later, and the old name would pop right back. My first instinct: the cache. I pulled up the monitoring dashboards. Redis memory looked fine, database connections steady as a rock. Yet users kept getting a snapshot from 12 hours ago. After six hours of log diving, I finally found the culprit: a 200ms concurrency window buried inside the “write DB then delete cache” logic of the Cache‑Aside pattern. Just long enough for a read request to sneak in, fetch the stale data from MySQL, and write it back into Redis before the delete could fire.

After the dust settled, I realized that hunting this down manually — combing through logs, eyeballing timestamps — was never going to scale. What I needed was an automated test suite that could simulate concurrency, inspect Redis state directly, and compare that state against the database like a forensic snapshot. So I built a solution using pytest + fakeredis + concurrency injection + snapshot assertions, and locked the whole class of cache‑consistency bugs inside a CI cage.

Why do cache‑consistency bugs only explode in production?

The scenario is classic. You have a User service: reads check Redis first, fall back to MySQL on a miss, and backfill the cache. Writes go to MySQL first, then delete the corresponding Redis key. It’s the textbook Cache‑Aside pattern.

Looks bulletproof on paper. But the devil lives in the concurrency. Imagine request A is an update, and B is a read:

A writes MySQL successfully — nickname changes from "Tom" to "Jerry".
Before A gets a chance to delete the cache, B arrives.
B queries Redis. The key has expired or doesn’t exist yet → cache miss.
B hits MySQL and reads "Jerry" (the new value). Lucky case? Sure, but what if the old value "Tom" was still sitting in cache before step 1? Then B returns the stale data straight from Redis.
The truly deadly interleaving, though, is this:
- Cache has no key at the start.
- A writes DB → B reads DB (getting the old value) → A deletes cache (a no‑op) → B writes the old value back into Redis. Result: the cache is now permanently stale until the next write or TTL eviction.

Even sneakier: in many codebases, the “delete cache” step isn’t atomic, or it gets swallowed by an exception handler. You end up with a database that’s completely up‑to‑date while the cache clings to yesterday’s data. You stare at the code for half an hour and see nothing wrong, but once QPS climbs, that probabilistic bug turns into a full‑blown outage.

Conventional testing — stepping through a debugger, sprinkling logs, manually flushing caches — is blind to these race windows. You need a way to reproduce concurrent execution orders in a test and then capture the full Redis state to make assertions.

Design choices: why an in‑memory snapshot instead of a real Redis instance?

In theory, you could spin up a real Redis instance and run integration tests against it. Anyone who’s tried knows it’s a nightmare:

Environment dependencies: Local dev and CI both need Redis installed, with matching versions — otherwise you’re debugging command differences instead of your app.
Data pollution: Multiple test cases share the same Redis. You must FLUSHALL before every test, but clean‑up is never perfect and parallel execution causes cross‑talk.
Slowness: Network I/O drags test speed from microseconds to tens of milliseconds. When you have hundreds of cases, you want to throw your laptop out the window.
Time‑travel difficulties: Testing TTL or eviction logic forces you to sprinkle time.sleep or manipulate the system clock — a time‑bomb for CI pipelines.

That’s why I chose fakeredis — a pure‑Python, in‑memory library that speaks the same redis-py commands. It lives inside the process, gets thrown away after each test, provides perfect isolation, and runs blazingly fast. More importantly, it gives us a natural entry point for an in‑memory snapshot: after the test executes, we can walk all keys, dump their values into a dict, and compare that dict against the expected database state. It’s like taking a crime‑scene photograph that preserves every piece of evidence.

Why not just mock the Redis calls? A mock verifies “did you invoke this command?”, but it can never verify whether the actual cache content is correct. Worse, every time the code adds a new SETEX, your carefully crafted assert_called_once_with lines break, sending maintenance costs through the roof. A memory snapshot doesn’t care what commands were called — it only looks at the final state. That’s the kind of testing that actually pays off.

Core implementation: from concurrency reproduction to snapshot evidence

1. First, build the caching layer with the concurrency bug built in

The code below demonstrates the problematic “bad case”. In a real project, this might sit inside your ORM or service layer; I’ve simplified it into a UserCache class:

import time
from typing import Optional
import redis  # used only for type hints; tests will swap in fakeredis

class UserCache:
    def __init__(self, db_conn, redis_conn):
        self.db = db_conn
        self.redis = redis_conn

    def get_user(self, user_id: int) -> Optional[dict]:
        key = f"user:{user_id}"
        data = self.redis.get(key)
        if data:
            return eval(data)  # 简化反序列化
        # cache miss -> 查库
        user = self.db.query_one("SELECT * FROM users WHERE id=?", (user_id,))
        if user:
            self.redis.set(key, str(user))
        return user

    def update_user(self, user_id: int, new_name: str):
        # 1. 更新数据库
        self.db.execute("UPDATE users SET name=? WHERE id=?", (new_name, user_id))
        # 时间窗口就在这里：下面的删缓存在“稍后”执行
        # 并发读可能在这中间把旧值写回 Redis
        self.redis.delete(f"user:{user_id}")

That window might only be a few microseconds in reality, but it’s wide enough to get hit under load. We need to precisely recreate it in a test.

2. Set up fakeredis + the snapshot foundation with pytest fixtures

This snippet solves two things: an isolated “fake Redis” for every test, and a utility for taking those forensic snapshots:

# conftest.py
import pytest
import fakeredis
from collections.abc import Mapping

@pytest.fixture
def fake_redis():
    """每个测试独立的 fakeredis 实例，省去 FLUSHALL"""
    server = fakeredis.Fa

AI Customer Service Lost 30% Memory in Week One — I Wrote 3 Pytest Tests to Catch the Culprit

BAOFUFAN — Fri, 26 Jun 2026 12:05:04 +0000

At 2 a.m. last Thursday, my ops colleague blew up my phone: “Customers are flooding the complaint channels. Our AI support bot has amnesia — one second a customer says ‘My order number is 8892’, the next the bot asks ‘May I have your order number?’” We had just launched a smart customer service system built with LangChain for the conversation chain. Load testing during the day looked perfect, but once real traffic hit at night, the memory loss rate spiked above 30%. I could feel the customers’ fury through the screen.

What chilled me most was this: if the same bug happened in a fintech or healthcare scenario, it wouldn’t be just complaints — it could be catastrophic. That same night I made the decision: we must lock down “memory storage consistency” with an automated test suite so it never slips into production again.

Breaking down the problem: why does LangChain’s memory just disappear?

Our support system looked roughly like this:

User -> FastAPI -> LangChain ConversationChain (with memory) -> LLM

To move fast during the initial launch, we chose ConversationBufferMemory with the default InMemoryChatMessageHistory. Under the hood, it’s essentially a dictionary: the key is the session_id and the value is a list of messages. It seemed fine — until we deployed with multi-process Gunicorn and multiple Pods. Three culprits then struck simultaneously:

No shared memory: A user’s first request lands on Pod A, which stores the memory in Pod A’s local memory. The second request gets load-balanced to Pod B, which has no memory at all → amnesia.
No concurrency control: When multiple coroutines or threads write to the same session dictionary concurrently, messages overwrite each other and memory becomes a garbled mess.
No persistence: When a Pod restarts, all customer memory evaporates. Returning customers instantly become “new users”.

Of course we knew we should switch to a shared store like Redis. But the truly fatal flaw was this: from development to testing, no step ever verified the correctness of memory storage. Business tests only ran single-threaded, single-process happy paths. Concurrency and restart scenarios were a complete blind spot. That’s the classic trap: you think everything is fine — until it blows up.

Designing the fix: why Pytest + Redis in three acts?

Plugging this hole required more than just swapping the backend to Redis — we needed tests that could truly nail down consistency. My goal was clear: write a set of Pytest tests that automatically verify three critical behaviors — isolation, persistence, and concurrency safety.

The tech choices were straightforward:

Redis as the memory backend: In production, multiple Pods inside K8s must share state. Redis is the infrastructure our team knows best, and it has the officially maintained RedisChatMessageHistory (in langchain-community).
Pytest + pytest-asyncio as the test framework: Our project is an async FastAPI service, so tests must be async too. pytest-asyncio makes it easy to simulate concurrent scenarios.
No mocking, real Redis connection: Memory consistency problems hide exactly in real network calls, serialization, and connection pooling. Mocking those would be self-deception. We use an isolated test Redis database (db=15) and flush it after each test — no interference.

Why not just add logging in production and observe? Because “occasionally lost” bugs drown in a sea of normal requests. By the time you locate the affected session, the customer is already gone. A test written before the incident is ten thousand times cheaper than firefighting afterward.

Core implementation: three tests that lock down memory consistency

1. First, set up the test infrastructure (conftest.py)

This code ensures every test gets a clean Redis memory backend:

# conftest.py
import pytest
import redis
from langchain_community.chat_message_histories import RedisChatMessageHistory

REDIS_URL = "redis://localhost:6379/15"  # dedicated test db

@pytest.fixture
def redis_client():
    """Create a Redis client, clear data after the test"""
    client = redis.Redis.from_url(REDIS_URL, decode_responses=True)
    yield client
    client.flushdb()  # clean isolation for each test
    client.close()

@pytest.fixture
def make_history():
    """Factory: create memory instance with a given session_id"""
    def _make(session_id: str) -> RedisChatMessageHistory:
        return RedisChatMessageHistory(
            session_id=session_id,
            url=REDIS_URL,
            ttl=3600  # matches production; expiration also causes "loss"
        )
    return _make

2. Test 1: Isolation — conversation A must never leak into conversation B

This test verifies that memories from different session_ids are completely isolated, catching low-level but deadly bugs like key concatenation errors or wrong Redis db writes.


python
# test_memory_isolation.py
import pytest

@pytest.mark.asyncio
async def test_memory_isolation(make_history):
    """Two different sessions should not see each other's messages"""
    history_a = make_history("session_a")
    history_b = make_history("session_b")

    # Simulate session A: customer mentions an order number
    await history_a.aadd_user_message("我的订单号是 8892")
    await history_a.aadd_ai_message("好的，正在为您查询订单 8892")

    # Simulate session B: completely different topic
    await history_b.aadd_user_message("我想退货")
    await history_b.aadd_ai_message("请提供您的手机号")

    # Assert: session A must have absolutely zero content from B
    msgs_a = await history_a.aget_messages()
    content_a = " ".join([m.content for m in msgs_a])
    assert "退货" not in content_a, "A 会话泄漏了 B 的对话内容！"
    assert "8892" in content_a, "A 会话自身的消

I Used pytest for 3 Years Before Realizing I Never Actually Tested LLM Memory Persistence

BAOFUFAN — Fri, 26 Jun 2026 01:03:25 +0000

It was 2:30 a.m. when a user complaint exploded on my phone: “Your chatbot has amnesia again—it forgot everything we talked about three days ago!” I scrambled out of bed, dug through the logs, and discovered that a seemingly harmless “optimize Redis serialization” PR from the previous week had silently caused the legacy message loader to throw an exception whenever it encountered the new structure. Worse, our live memory storage module had zero coverage for that regression path. The irony? I had been writing pytest for three years, but every single test mocked Redis, mocked the storage layer, and only exercised business logic. I had never actually verified memory persistence and consistency regression against a real database. After that night, I changed everything.

This article is my blood-and-tears postmortem: how to build a reproducible, regression-proof integration test suite for LLM memory storage with pytest and Docker—so that “what you stored yesterday can be read back exactly today, and it survives a container restart tomorrow.”

Breaking down the problem: why mocks can’t save memory storage

LLM memory storage sounds trivial: save conversations, load conversations. But in the real world, three tricky dimensions immediately surface:

Version compatibility of serialization/deserialization

You store a Message object, but when you read it back, you deserialize it into a dict or a Pydantic model. The moment a timestamp format, a role enum value, or an extra metadata field changes, a mock-based test that fakes disk reads and writes will never notice—because the mock always returns idealized data you prepared in advance.
Concurrency consistency

A user can fire off several messages in rapid succession within the same session, triggering concurrent writes. Redis’s HSET and ZADD are atomic, but if you wrap them in a “read-then-write” merge step (e.g., merging a context window), you can easily get overlapping updates and lost messages. A pure unit test running single-threaded will never hit such race conditions, not even after 100 runs.
Persistence regression

After a Docker container restart, can the AOF/RDB files on the mounted volume restore everything correctly? Are you relying on Redis’s default save configuration, or did you happen to land exactly within the 900-second window where no snapshot triggered? If an ops colleague casually bumps the Redis image version or tweaks the persistence policy, previous conversations can vanish into thin air. Unless you actually kill the container and then read the data back, this pitfall stays forever in your blind spot.

The root cause is simple: memory problems are always compound failures of storage engine × access pattern × time. Mocks amputate the storage engine, and manual testing removes the high-frequency repetition—you will never wait long enough for the failure to surface.

Designing the solution: real dependencies with Docker, sharp regression with pytest

We need a real storage environment that can be created and destroyed on demand, and we need to run the exact same set of consistency and persistence test cases against it over and over.

Comparing the candidates:

Option A: Point your dev environment directly at a local Redis. Downside: local data gets polluted, you have to manually flush between tests, and your colleague’s Redis version may differ from yours—no guaranteed regression consistency.
Option B: Bring up the full dependency stack with docker-compose and run pytest by hand. Downside: extra process management, not atomic enough, and tricky to achieve unit-test-level retries in CI.
Option C (the one I settled on): testcontainers-python to spin up Redis/Postgres containers on-demand inside pytest fixtures, automatically destroyed when the test finishes. Combine that with pytest’s scope and autouse to control container lifetimes, achieving isolation while keeping startup costs reasonable.

Why not pytest-docker? It’s better suited for running a long-lived container that you start once in CI. testcontainers, on the other hand, lets you treat the database as part of the test itself, dynamically created. When writing tests, you don’t have to care whether the external environment is already prepared, and parameterization becomes far easier—e.g., running the same test against Redis 6 and Redis 7.

Architecturally, we define an abstract MemoryStore base class. If we later swap in Postgres/pgvector, we can reuse the exact same consistency test suite. The key principle: tests don’t care about implementation details; they care about the contract. What you write is what you should read back; after a restart, historical data must not be lost.

Core implementation: inviting the Redis container into a pytest fixture

1. Abstract memory storage interface

This snippet decouples the tests from the concrete storage implementation—we only need to define the contract for reading, writing, and persistence behavior.

# llm_memory/store.py
from abc import ABC, abstractmethod
from typing import List, Optional
from dataclasses import dataclass, asdict
from datetime import datetime

@dataclass
class Message:
    role: str          # 'user' | 'assistant' | 'system'
    content: str
    timestamp: float   # epoch second, 保证跨语言兼容

class MemoryStore(ABC):
    @abstractmethod
    def append_message(self, session_id: str, msg: Message) -> None:
        ...

    @abstractmethod
    def get_history(self, session_id: str, limit: int = 100) -> List[Message]:
        ...

    @abstractmethod
    def close(self) -> None:
        ...

2. RedisMemoryStore implementation—must handle serialization edges

This piece addresses real-storage serialization and deserialization consistency. We deliberately use JSON instead of pickle because pickle has terrible cross-version compatibility and is unsafe.

# llm_memory/redis_store.py
import json
import redis
from typing import List
from llm_memory.store import MemoryStore, Message

class RedisMemoryStore(MemoryStore):
    def __init__(self, host: str, port: int, db: int = 0):
        self.client = redis.Redis(host=host, port=port, db=db,
                                  decode_responses=True)

    def _session_key(self, session_id: str) -> str:
        return f"session:{session_id}:messages"

    def append_message(se

DEV Community: BAOFUFAN

How We Caught 90% of Our LLM Memory Bugs with Playwright

Breaking down the problem: how memory silently disappears

Why Playwright for end-to-end memory verification?

Core implementation: let Playwright repeat every bizarre user action

1. Setting up a repeatable test environment

2. Simulating multi-turn conversation and asserting memory retention

3. Testing memory recovery after refresh — the real danger zone

4. Verifying localStorage contents directly — catching hidden corruption

The real culprit: serialization silently breaks your context

Takeaways: make memory testing a habit, not an afterthought

Chasing Chroma’s Silent Data Loss: How Playwright E2E Tests Saved 72 Hours

Breaking down the problem

Designing the solution

Core implementation

12 Hours Debugging Memory Drift in a LangChain+Chroma Code QA Bot

Problem Deconstruction

Solution Design

Core Implementation

1. Code Indexing into Chroma (persisted locally)

2. Q&A Chain with Dual-Layer Memory

Redis Persistence Pitfalls: A Data Loss Bug That Took Me 6 Hours to Uncover

Breaking Down the Problem: Why Does Data Still Disappear When Persistence Is Enabled?

Designing the Solution: Parameterized Validation with Pytest and Docker

Core Implementation: Building the Automated Validation Step by Step

LangChain Chatbot Session Mix-Up: When 100 Concurrent Users Got Each Other's Replies

Breaking It Down: Why User Sessions “Cross Wires”

Design Decisions: Why We Didn’t Pick the Alternatives

Core Implementation: Three Steps to Isolation

Redis Crash Recovery Testing: The 2 Bugs Manual Testing Missed

Problem Breakdown — Why Manual Testing Misses "Selective Forgetting"?

Solution Design — Why Pytest + Testcontainers?

Core Implementation — Building Memory Consistency Tests Step by Step

Dependency Installation and Container Fixture (conftest.py)

After 3 Years of Mocking Redis, We Missed 60% of Edge Cases in Memory Store Tests

Why Mocked Tests Make Your Redis Memory Storage a Paper Tiger

Why Testcontainers Instead of In‑Memory Redis or Shared Environments

Core Implementation: From Spinning Up a Container to Deterministic Integration Tests

1. A Reusable Redis Container Fixture

2. A Real, Disk‑Backed Memory Storage Class

3. Integration Tests That Cover the 60% Hidden Edge Cases

The “No Excuses” CI Strategy: Speed Meets Reality

Closing: Replace the Fake Green Light with Real Confidence

How I Fixed LangChain’s Agent Memory Loss with 23 Pytest Cases

Where exactly does an agent’s memory get lost?

Design decisions: Why Pytest + fakeredis instead of a real Redis cluster

Core implementation: from fixtures to multi-turn conversation tests

Setting up an isolated memory store so every test starts fresh

Verifying the most critical behavior: you can save and retrieve what you stored

Simulating real-world multi-turn conversations concurrently

Testing what happens when Redis misbehaves

Results and takeaways

From 2 Hours to 10 Minutes: Pytest Parametrization for LLM Memory Tests

Breaking Down the Problem

Why Conventional Approaches Fail

Solution Design

Core Implementation

Caging Redis Cache Inconsistency: A 6-Hour Debugging Nightmare Solved with pytest and In-Memory Snapshots

Why do cache‑consistency bugs only explode in production?

Design choices: why an in‑memory snapshot instead of a real Redis instance?

Core implementation: from concurrency reproduction to snapshot evidence

1. First, build the caching layer with the concurrency bug built in

2. Set up fakeredis + the snapshot foundation with pytest fixtures

AI Customer Service Lost 30% Memory in Week One — I Wrote 3 Pytest Tests to Catch the Culprit

Breaking down the problem: why does LangChain’s memory just disappear?

Designing the fix: why Pytest + Redis in three acts?

Core implementation: three tests that lock down memory consistency

1. First, set up the test infrastructure (conftest.py)

2. Test 1: Isolation — conversation A must never leak into conversation B

I Used pytest for 3 Years Before Realizing I Never Actually Tested LLM Memory Persistence

Breaking down the problem: why mocks can’t save memory storage

Designing the solution: real dependencies with Docker, sharp regression with pytest

Core implementation: inviting the Redis container into a pytest fixture

1. Abstract memory storage interface

2. RedisMemoryStore implementation—must handle serialization edges

4. Verifying `localStorage` contents directly — catching hidden corruption