After 2 Years with LangChain, I Realized I’ve Been Testing Memory the Wrong Way

At 2 a.m., my phone rang—ops was on the line. The production chatbot was “forgetting” again. A user had just given their order number, yet two turns later the bot asked, “Could you please provide your order number?” The user started cursing. I stared at LangChain’s Memory code for a long time; the logs looked fine. Finally, I blurted out: “Have we ever written a real automated test for the memory?”

The answer was no. Up to that point, our testing consisted of manually opening a terminal, chatting with the bot, and eyeballing whether the replies carried over earlier context. That “human smoke test” works when the logic is simple, but as soon as you add conditional branches, chain multiple memory objects, or deal with async concurrency, the memory state becomes a Schrödinger’s box—perfect when you don’t test it, broken the moment you look closely.

This article records how I fully automated multi-turn conversation memory validation with Pytest, and the pitfalls LangChain’s official docs won’t tell you about.

Breaking Down the Problem

LangChain’s memory is essentially a stateful key-value store. At the end of each turn, the chain feeds the current input and output to the memory class; at the start of the next turn, the memory stitches history into a string and injects it into the prompt. The trouble lives between “stitching” and “storing”: if the memory’s internal state isn’t updated in time, or the formatting logic doesn’t match what was stored, you end up with a situation where the data is saved but the assembled prompt is wrong.

For example, we want the memory to contain only user and AI messages, but some built-in memory classes write SystemMessage or template placeholders into chat_memory as well. That leads to duplicate instructions in the next prompt, and the model goes haywire. You’d never catch this kind of bug by chatting manually, because you need to compare the exact sequence of stored messages with the sequence that gets loaded next turn.

Unit-testing the memory object alone isn’t enough, either. When LangChain’s chain executes predict, it first calls memory.load_memory_variables, then assembles the prompt, calls the LLM, and finally calls memory.save_context. If you bypass the chain and test the memory directly, you skip the most dangerous serialization/deserialization step. The bug will still blow up in production.

So we need an integration test: simulate multi-turn conversations through the full chain pipeline, capture the internal state after each save_context, and precisely compare it against expected values.

Designing the Solution

Tech stack: parameterized Pytest + FakeLLM + LangChain Callbacks.

Why not just mock the LLM and assert on the return value? Because memory consistency checks are about state transitions, not output text. After every turn we need to peek directly into the memory’s chat_memory.messages list and confirm it matches our expected conversation history.

Why not read the memory attribute exposed by ConversationChain and call it a day? The official load_memory_variables returns a formatted string—a second-hand product. The real raw messages may have been tampered with. So we bypass the formatting and access memory.chat_memory.messages directly.

The architecture is simple: a Pytest fixture builds a ConversationChain armed with a FakeListLLM, and a parameterized test function receives a list of multi-turn tuples (user input, fake AI reply). After each chain.predict, we immediately assert that memory.chat_memory.messages matches the expected history—role types and message count included.

Core Implementation

First, a fake LLM that never hits an HTTP endpoint

This solves the “you can’t rely on a real model during tests” problem and lets us control the AI’s response for each turn.

from typing import List
from langchain_core.language_models.llms import LLM
from langchain_core.outputs import Generation, LLMResult

class FakeListLLM(LLM):
    """Returns predefined responses in order, simulating multi-turn conversations."""
    responses: List[str]
    i: int = 0                   # current turn index

    def _call(self, prompt: str, stop=None, **kwargs) -> str:
        # Consume one response per call, simulating AI replies
        if self.i >= len(self.responses):
            self.i = 0           # loop if exceeded (adjust as needed)
        resp = self.responses[self.i]
        self.i += 1
        return resp

    @property
    def _llm_type(self) -> str:
        return "fake-list"

    def _generate(self, prompts: List[str], stop=None, **kwargs) -> LLMResult:
        # Provide _generate to keep LangChain’s internal call chain happy
        generations = [[Generation(text=self._call(p))] for p in prompts]
        return LLMResult(generations=generations)

A test fixture that lets us grab the memory’s raw state

This fixture returns both the chain and a reference to the memory, so we can inspect messages after every turn without waiting for the whole test to finish.

import pytest
from langchain.memory import ConversationBufferMemory
from langchain.chains import ConversationChain

@pytest.fixture
def chain_with_memory():
    """Returns the chain and memory reference for on-the-fly inspection."""
    memory = ConversationBufferMemory(return_messages=True)
    llm = FakeListLLM(responses=[])   # responses injected by the test
    chain = ConversationChain(
        llm=llm,
        memory=memory,
        verbose=False
    )
    # Return memory and llm together so tests can manipulate them directly
    return chain, memory, llm

Parameterized multi-turn conversation test

This part solves the core need: automatically iterate through N turns and assert that the memory state is always correct.

(原文此处为描述，未提供代码块，翻译时保留)

The test uses @pytest.mark.parametrize to feed in conversation scripts. Each script is a list of (user_input, expected_ai_output). Before the test, we set llm.responses to the list of expected AI outputs. Then we loop through the turns: for each user_input, call chain.predict(input=user_input) and immediately assert:

• The number of messages in memory.chat_memory.messages equals 2 * turn_index (one human + one AI per turn).
• The last two messages have the correct type (HumanMessage / AIMessage) and content.
• The entire message list exactly matches a pre-built expected list.

This approach catches bugs like missing save_context calls, incorrect message ordering, or rogue system messages leaking into the history. When the assertion fails, Pytest prints the full messages list, making it trivial to spot the exact turn where memory went off the rails.

Through dozens of runs, this setup has exposed subtle issues that manual testing never could—like ConversationSummaryBufferMemory pruning messages but still referencing older facts in its summary, or asynchronous chains where save_context races with the next load_memory_variables. Once you have deterministic assertions on the raw message list, those mysteries vanish.

If you’re using LangChain in production and still manually checking memory, consider giving this pattern a try. The confidence it brings is worth every line of test code.