Automating LLM Memory Validation with Pytest & Redis: 45x Faster Regression Testing

At 2 a.m., a colleague dropped a screenshot into our work chat — a user complaint: “It forgot who I was mid-conversation.” Staring at the logs, I found the session history had been written just fine, but the next request pulled back an empty memory. That was the moment I knew: if we didn’t automate regression testing for our memory storage, these failures would keep popping up like a never-ending game of whack-a-mole.

Breaking down the problem

LLMs are stateless by nature. All “memory” depends on us storing conversation history in an external store — usually Redis — keyed by session ID. On every request, we load that history, stitch it into the prompt, and hope it’s intact. If the storage logic has a bug (a misspelled key, mismatched serialization, or wrong TTL), the context is silently dropped. The LLM won’t error out. It will simply respond as if it’s never met you.

Previously, we relied entirely on manual regression: before each release, testers would manually create several sessions, run three to five turns of conversation, inspect the raw data in Redis field by field, then simulate the next request to check if the full context came through. One scenario took at least 2 minutes; 50 scenarios ate up nearly two hours. Worse, eyeballing JSON diffs makes it far too easy to miss subtle mismatches — missing regressions were basically inevitable. We desperately needed an automated suite that could simulate complete multi-turn conversations, make precise assertions against Redis memory, and do it without hitting real LLM APIs (which would be slow, flaky, and costly).

Design decisions

Why not a database? Because conversational latency is critical — every memory read/write must complete in milliseconds. Redis is the only realistic option. For testing, we naturally chose pytest. The whole team already knows it, and its plugin ecosystem is unbeatable.

I set three architectural rules:

1. Abstract a MemoryService that encapsulates all Redis read/write logic for conversation history. It only cares about two operations: “append a message” and “load context”. No model inference enters this layer.
2. Use fakeredis for local tests so the default runs entirely in memory, with zero external dependencies. Only integration tests should connect to a real Redis instance.
3. Mock the LLM calls so the tests purely validate memory correctness — no network jitter, no token consumption.

Why not test against the real LLM? Because what we need to guarantee is “context wasn’t lost”, not “the model answered well”. If the stored history can be retrieved exactly as it was written, we’ve already plugged 90% of the risk. Real LLM interaction is reserved for a handful of smoke-level end-to-end tests, covering just a few labelled scenarios.

Core implementation

1. Memory service: encapsulated Redis reads/writes with consistent serialization

The code below defines MemoryService. It appends each new message to a Redis list for the session, and when loading, it fetches the most recent N messages to form the context. We serialize with json for readability and easy debugging.

import json
from typing import List, Dict
from redis import Redis

class MemoryService:
    def __init__(self, redis_client: Redis, max_history: int = 20):
        self.redis = redis_client
        self.max_history = max_history

    def _session_key(self, session_id: str) -> str:
        return f"chat:session:{session_id}:history"

    def add_message(self, session_id: str, role: str, content: str) -> None:
        """追加一条消息到会话历史"""
        key = self._session_key(session_id)
        msg = json.dumps({"role": role, "content": content})
        self.redis.rpush(key, msg)
        # 保留最近 max_history 条，避免无限增长
        self.redis.ltrim(key, -self.max_history, -1)

    def get_context(self, session_id: str) -> List[Dict[str, str]]:
        """加载最近的对话上下文"""
        key = self._session_key(session_id)
        raw_messages = self.redis.lrange(key, 0, -1)
        return [json.loads(m) for m in raw_messages]

2. pytest fixture: inject fakeredis automatically, keep tests isolated

To guarantee each test case starts with a clean slate, we provide a memory_service fixture in conftest.py that defaults to fakeredis. Data is flushed after each test so they never interfere.

# conftest.py
import pytest
from fakeredis import FakeRedis
from memory_service import MemoryService  # 上面的模块

@pytest.fixture
def memory_service():
    """提供基于 fakeredis 的 MemoryService，测试完全隔离"""
    fake_redis = FakeRedis()
    service = MemoryService(fake_redis, max_history=10)
    yield service
    fake_redis.flushall()

3. Test cases: fresh sessions, multi-turn preservation, truncation, and concurrency

Now the tests themselves. The very first one verifies that a brand-new session carries no history. It looks trivial, but it’s critical — if an empty session returns dirty data, every subsequent assertion becomes meaningless.

def test_new_session_has_no_context(memory_service):
    ctx = memory_service.get_context("session_1")
    assert ctx == [], "新会话应该是空的，没有任何历史消息"

Then the core scenario: does multi-turn memory accumulate message by message, in the correct order?

def test_multi_turn_memory_preserved(memory_service):
    sid = "session_2"
    # 模拟用户与模型两轮对话
    memory_service.add_message(sid, "user", "我叫张三")
    memory_service.add_message(sid, "assistant", "你好张三，有什么事？")
    memory_service.add_message(sid, "user", "我刚才说我叫什么？")

    context = memory_service.get_context(sid)
    assert len(context) == 3
    assert context[0]["role"] == "user"
    assert context[0]["content"] == "我叫张三"
    assert context[2]["content"] == "我刚才说我叫什么？"

Our MemoryService caps the number of stored messages. We must also ensure the oldest messages are trimmed correctly.

def test_history_truncation(memory_service):
    sid = "session_3"
    # 写入 12 条消息，max_history=10，最早的两条应该被丢弃
    for i in range(12):
        memory_service.add_message(sid, "user", f"msg_{i}")

    context = memory_service.get_context(sid)
    assert len(context) == 10
    # 第一条保留的应该是 msg_2（index 2）
    assert context[0]["content"] == "msg_2"
    assert context[-1]["content"] == "msg_11"

Because Redis is inherently single-threaded for command execution (in these testing scenarios we don’t need to stress actual concurrency), we can also simulate rapid sequential writes and verify that no messages are lost.

def test_rapid_sequential_writes(memory_service):
    sid = "session_4"
    messages = []
    for i in range(50):
        role = "user" if i % 2 == 0 else "assistant"
        content = f"message_{i}"
        memory_service.add_message(sid, role, content)
        messages.append({"role": role, "content": content})

    stored = memory_service.get_context(sid)
    # 因为 max_history=10，我们只保留了最后10条
    expected = messages[-10:]
    assert stored == expected

These cases already cover the most dangerous blind spots: empty state, history buildup, truncation, and write-under-load. All run in under a second total, with zero remote dependencies.

Real-Redis integration tests (when needed)

For scenarios that truly require real Redis — cluster mode, eviction policies, or Lua scripts — we add a pytest marker and a separate fixture that connects to a test instance. It’s guarded so it never runs by default.

@pytest.fixture
def real_redis_memory():
    import os
    host = os.getenv("TEST_REDIS_HOST", "localhost")
    port = int(os.getenv("TEST_REDIS_PORT", 6379))
    client = Redis(host=host, port=port, db=15)
    service = MemoryService(client, max_history=10)
    yield service
    client.flushdb()

Results and impact

Before this refactor, running the full memory-regression checklist took around 90 minutes of manual work per release cycle. With the pytest suite, the same 50 scenarios (and more) execute in under two minutes, giving us a 45x speedup. More importantly, the tests are deterministic and independent of human attention — they catch missing context 100% of the time.

We integrated the suite into our CI pipeline. Now every commit triggers these checks, and any silent context loss is flagged before it can ever reach a user. The “it forgot me” bug hasn’t resurfaced since.

If your team is still manually verifying conversational memory in an LLM-powered product, a similar approach will give you both velocity and confidence. Start by wrapping your Redis logic in a thin service, inject fakeredis, and write the tests that match your worst nightmares. The time you invest will pay back every single regression cycle.