At 1 a.m., our user group blew up: “Is your AI a goldfish? I just told it my name is Wang Jianguo, and in the next turn it asks ‘What’s your name?’” I tested it dozens of times locally using Swagger, and it remembered every time. Until I wrote a test suite with Pytest+Docker that simulates cross-session reconnects — 30% of the cases instantly turned red. That was the moment I realized: all that “manual verification” was just a mental comfort blanket.
Breaking Down the Problem: Why Manual Testing Misses Cross-Session Context Loss
The typical AI memory scenario goes like this: a user starts a session (Session A) and says, “My name is Wang Jianguo, and I like playing basketball.” Later that same day, they reconnect in a new session (Session B). The AI should be able to say, “Wang Jianguo, there’s an update on the basketball tournaments you follow.” If the AI instead asks, “What’s your name?”, you have a context loss.
The root cause often isn’t the model itself, but the memory storage implementation. For instance, you store conversation history in Redis, but the key design depends on the session ID without merging at the user level. Or you set an expiration time, so the memory gets evicted when the session disconnects. Or even worse: with multiple instances, caches aren’t synchronized, so when reconnecting you get routed to a pod that has no memory.
The fatal flaw of manual testing is that you always chat from start to finish in a single browser tab, never simulating real-world edge cases like reconnections, multi-device switching, session ID changes, and cache eviction. Your brain automatically fills in the context, but automated assertions don’t.
Solution Design: Pytest + Docker Builds a Reproducible Memory Stress-Test Environment
To reliably catch context loss, you need two things: a real storage environment and stateless session orchestration.
On the storage side, I run Redis 7 in Docker because in production, AI memory is mostly persisted with Redis or vector databases. No in-memory dict — that thing never loses data and hides all the bugs. No mocks — mocking reads and writes without network jitter means production will still explode.
For testing, I use Pytest because its fixture mechanism elegantly manages Docker container lifecycles, ensuring each test starts from a clean, isolated state. The orchestration plan:
- Use
docker-composeto start Redis, exposing the port. - In Pytest’s
conftest.py, define amemory_servicefixture that connects to this Redis and returns a wrapped memory service object. - Test cases simulate reads and writes for the same user but with different session IDs, asserting that memory survives across sessions.
- Introduce
time.sleepor proactiveexpireto verify TTL eviction scenarios.
Core Implementation: From Zero to Running
The snippet below wraps a Redis-based AI memory service, solving memory storage and user-level merging.
# ai_memory/memory_service.py
import json
from typing import List, Dict, Optional
import redis
class MemoryService:
"""基于Redis的AI记忆服务，按user_id归并历史，而非session_id"""
def __init__(self, redis_client: redis.Redis, ttl: int = 3600):
self.redis = redis_client
self.ttl = ttl # 记忆过期时间，默认1小时
def _key(self, user_id: str) -> str:
# 关键设计：key只跟用户有关，与session_id解耦
return f"memory:{user_id}"
def append(self, user_id: str, message: Dict) -> None:
"""追加一条对话记录"""
key = self._key(user_id)
history = self.get_history(user_id)
history.append(message)
# 序列化后写回，并重置TTL
self.redis.set(key, json.dumps(history, ensure_ascii=False), ex=self.ttl)
def get_history(self, user_id: str) -> List[Dict]:
"""获取用户完整历史"""
key = self._key(user_id)
raw = self.redis.get(key)
if raw is None:
return []
return json.loads(raw)
def save_context(self, user_id: str, context: Dict) -> None:
"""保存结构化上下文（如姓名、偏好）"""
ctx_key = f"context:{user_id}"
self.redis.set(ctx_key, json.dumps(context, ensure_ascii=False), ex=self.ttl)
def get_context(self, user_id: str) -> Optional[Dict]:
raw = self.redis.get(f"context:{user_id}")
if raw:
return json.loads(raw)
return None
This conftest ensures Docker Redis is ready before tests, solving environment consistency.
# tests/conftest.py
import pytest
import redis
import subprocess
import time
import os
@pytest.fixture(scope="session")
def docker_redis():
"""启动docker-compose中的Redis，测试结束后销毁"""
compose_file = os.path.join(os.path.dirname(__file__), "docker-compose.yml")
subprocess.run(["docker-compose", "-f", compose_file, "up", "-d"], check=True)
# 等待Redis完全就绪，否则会连不上
time.sleep(2)
yield
subprocess.run(["docker-compose", "-f", compose_file, "down", "-v"], check=True)
@pytest.fixture
def me
Top comments (0)