At 4:59 PM on a Friday, I was about to close my laptop and sneak out when the QA colleague's icon flashed on DingTalk: "Come check this out. The support bot remembers I'm Zhang San, but when I ask for my order number, it insists it belongs to Li Si." I pulled up the logs and saw LangChain's ConversationBufferMemory behaving like it had severe amnesia — Session A was mixing up chat history from Session B. In that moment, I knew that unless I built an automated test suite to lock down the accuracy and consistency of memory storage, the next blow-up would definitely happen at 2 AM.
Breaking Down the Problem
In LLM-powered chat products, the memory module is responsible for remembering context across multiple turns — so that when the user said earlier "I live in Beijing," the weather query later can automatically include "Beijing." Sounds easy, but things get messy once you land in LangChain: ConversationBufferMemory stores all conversations in plain text. It works fine as long as the memory fits in RAM, but switch to Redis or a database for persistence, and a whole bunch of issues bubble up — serialization/deserialization, concurrent reads/writes, and trimming old messages.
In our production scenario, a customer service bot handled hundreds of concurrent users. Each user session was independent but they all shared a common Redis instance. When we first launched, QA manually tested a dozen typical conversation paths and found absolutely no cross-session memory leaks, because manual testing simply can't cover race conditions under high concurrency, nor reproduce edge cases where trim_messages mixes up adjacent sessions when a Redis connection blinks out. Once real traffic hit, bugs popped up like whack-a-mole — you fixed one, another sprang out. We desperately needed a set of regression tests that could directly verify memory read/write accuracy and cross-session isolation.
Designing the Solution
The goal was clear: run through the core logic of the memory module right in our local CI, without a real LLM or a real Redis instance, and catch issues before any code landed.
Framework choice was a no-brainer — Pytest. Its fixture capabilities are perfect for assembling different memory instances. LangChain's memory abstraction is fairly clean: BaseChatMemory provides uniform save_context and load_memory_variables interfaces, so we could write the same set of tests against different memory backends. A real Redis is too heavy, so we chose fakeredis to simulate a Redis instance in memory — quick to spin up and zero side effects. All LLM calls were banished with unittest.mock, because we were testing memory, not the LLM.
Why not use the built-in langchain.tests? They only cover the shallowest of interfaces, none of the hard-won scenarios like message type conversion or multi-session isolation. We also didn't want to run Redis in a Docker container — our CI resources are already stretched thin; adding one more container would jam the build queue by an extra 3 minutes.
The overall architecture: define a fake_redis_memory fixture inside Pytest's conftest.py, use it to construct different Memory subclasses (ConversationBufferMemory, ConversationSummaryMemory), simulate multi-turn conversations with helper functions, and then assert that the history returned by load_memory_variables is both complete and free of cross-session contamination.
Core Implementation
1. Building a Zero-Dependency Test Harness
This snippet packages fakeredis, mock LLM, and Memory instantiation into a fixture. All subsequent test cases run on top of it. The non-negotiable requirement: zero network requests, and any single test completes in under 0.3 seconds.
# conftest.py
import pytest
from unittest.mock import MagicMock
from langchain.memory import ConversationBufferMemory
from langchain_community.chat_message_histories import RedisChatMessageHistory
from fakeredis import FakeRedis
@pytest.fixture
def fake_redis_memory():
# 用 fakeredis 构建一个假 Redis 客户端
fake_redis_client = FakeRedis()
def _create_memory(session_id: str):
# 注入伪造的 Redis，保证每次测试的 session 隔离
history = RedisChatMessageHistory(
session_id=session_id,
redis_client=fake_redis_client
)
# ConversationBufferMemory 默认 return_messages=True 时，会返回 Message 对象
memory = ConversationBufferMemory(
chat_memory=history,
return_messages=True # 关键：确保拿到结构化消息，方便断言
)
return memory
return _create_memory
2. Testing Accuracy: Every Message Written Must Come Back
This test simulates two rounds of conversation and verifies that the history returned by load_memory_variables has the exact length and content we expect. It puts an end to the mysterious "I stored two lines but only got one back" bug.
python
# test_memory_accuracy.py
from langchain.schema import HumanMessage, AIMessage
def test_buffer_memory_keeps_all_messages(fake_redis_memory):
memory = fake_redis_memory("session_1202")
# 模拟第一轮对话
memory.save_context(
{"input": "我叫张三"},
{"output": "你好张三"}
)
# 模拟第二轮对话
memory.save_context(
{"input": "我的订单号是多少"},
{"output": "你的订单号是 #1123"}
)
variables = memory.load_memory_variables({})
history = variables.get("history", [])
# 断言：总共应该有 4 条消息（两问两答）
assert len(history) == 4
assert isin
Top comments (0)