LangChain Memory Testing Pitfalls: The Two Bugs That Cost Me 4 Hours

At 1 AM, the customer group blew up — our AI support agent suddenly developed amnesia. A user had just given their order number, and in the very next turn the bot asked, "May I have your order number, please?" After digging through layers of code, I found that the memory storage logic had been accidentally changed during a refactor. Right then I made up my mind: memory must be tested. What I didn’t expect was that testing LangChain’s memory components with Pytest would swallow yet another entire afternoon. This post captures two real-world traps so you don’t fall into them.

Why Memory Testing Is Tricky

Conversational coherence in AI agents depends entirely on memory. LangChain ships with ConversationBufferMemory, ConversationSummaryMemory, and several other implementations, but most of the time we only use them indirectly through a Chain. When something breaks, we manually simulate a few dialogue turns and visually check if the response contains historical information. Manual testing has two fatal flaws:

1. High repetition cost: Every time you change the code, you have to replay multi-turn conversations from scratch and compare replies by eye.
2. Leaky state: Memory is stateful. One test run’s conversation leaks into the next, so you never know if a failure comes from broken memory logic or from leftovers you forgot to clean.

What we really need is a repeatable, auto-asserting test suite that directly verifies the memory component’s output across different conversation sequences — not an end-to-end lottery with the whole Chain.

Design: Test the Memory, Not the Chain

I landed on Pytest + LangChain’s Memory interface, with the core idea of testing the memory object in isolation.

• Why Pytest?
  
  pytest.mark.parametrize is perfect for running the same test suite across different memory types. fixtures elegantly manage the lifecycle of memory instances so state doesn’t bleed between tests.

• Why test Memory directly instead of the whole Chain?
  
  The Chain also includes an LLM, which is slow and unstable. Memory is essentially a pure-logic module that extracts context from conversation history. Unit-testing its load_memory_variables output is enough. If that works and we mock the LLM, the layer above can’t break.

• Why not Unittest?
  
  I need to create and destroy memory instances frequently. Pytest’s fixture mechanism is more flexible than setUp/tearDown, and parameterization lets me cover six memory types with one command, saving piles of boilerplate.

Core Implementation: Automated Memory Tests from Scratch

1. Testing ConversationBufferMemory – Verify What Was Remembered

This test exercises the simplest buffer memory, simulating one user message and then asserting the returned context contains that exact utterance.

import pytest
from langchain.memory import ConversationBufferMemory

# fixture 保证每个测试拿到全新的记忆实例
@pytest.fixture
def buffer_memory():
    # return_messages=False 让 load_memory_variables 返回纯字符串，方便断言
    return ConversationBufferMemory(return_messages=False)

def test_buffer_memory_remembers_input(buffer_memory):
    """一轮对话后，记忆应包含用户输入"""
    # 模拟一轮完整的对话上下文
    buffer_memory.save_context(
        {"input": "我的订单号是ABC123"},
        {"output": "好的，已为您查询到订单信息"}
    )
    # 提取记忆变量
    memory_vars = buffer_memory.load_memory_variables({})

    # 断言历史字符串包含用户原话
    assert "ABC123" in memory_vars["history"]
    assert "Human: 我的订单号是ABC123" in memory_vars["history"]

2. Testing ConversationSummaryMemory – Verify That a Summary Gets Generated

Summary memory relies on an LLM to compress history. To keep tests fast we mock the LLM. Here I use LangChain’s FakeListLLM to return a fixed summary.

from langchain.memory import ConversationSummaryMemory
from langchain.llms.fake import FakeListLLM

@pytest.fixture
def summary_memory():
    # Fake LLM 直接返回预设摘要，不依赖真实模型调用
    llm = FakeListLLM(responses=["用户询问了订单ABC123的情况"])
    return ConversationSummaryMemory(llm=llm, return_messages=False)

def test_summary_memory_returns_fake_summary(summary_memory):
    summary_memory.save_context(
        {"input": "我的订单号是ABC123"},
        {"output": "查询中..."}
    )
    memory_vars = summary_memory.load_memory_variables({})
    # 摘要应为 Fake LLM 返回的内容
    assert "用户询问了订单ABC123的情况" in memory_vars["history"]

This runs in milliseconds — no OpenAI API call required.

Pitfalls: Two Traps That Ate 4 Hours of My Life

Pitfall 1: return_messages=True Breaks All Your String Assertions

What happened:

I set return_messages=True on ConversationBufferMemory and then confidently wrote a bunch of assert "some word" in memory_vars["history"]. The tests all went red. Printing memory_vars["history"] showed something like [HumanMessage(content='...'), AIMessage(content='...')] — a list of objects, not a string.

Root cause:

When return_messages=True, the memory returns conversation history as LangChain message objects (e.g., HumanMessage, AIMessage) instead of a pre-concatenated text. The load_memory_variables output then contains that list of message objects, which of course makes any in-on-string assertion fail.

Fix:

Either keep return_messages=False for tests that rely on string matching, or update your assertions to work with message objects:

# 如果启用了 return_messages=True，应当这样断言
history = memory_vars["history"]
assert any("ABC123" in msg.content for msg in history)

Always confirm the data type of history before you write assertions. A quick print(type(memory_vars["history"])) would have saved me an hour.