Contract Testing Vector DB Memory Storage with pytest + Allure: 60% Fewer Regression Incidents

At 2 a.m., I was woken by an alert: the memory storage of our production LLM assistant had returned allergy information that a user had deleted three days earlier, causing it to recommend food containing peanuts. After half a night of investigation, we found that after a vector database SDK upgrade, the default behavior of delete changed from “immediately effective” to “eventually visible”. That night marked the end of our team’s era of manual curl verification.

Problem Breakdown

When using a vector database for LLM memory storage, there are only three core operations: upsert (write/update memory), query (similarity search), and delete (remove expired memory). However, the semantics of these operations can differ subtly across versions and deployment modes (local in-memory, Docker, cloud services). For example, after an upsert updates the same ID, can the old document still be retrieved? Is delete synchronous or asynchronous? When a filter condition is empty, can it accidentally wipe out the entire collection?

Our team previously had no automated tests. Before each release, developers manually ran a few Python scripts or used curl to hit HTTP endpoints. As a result, SDK upgrades, server-side configuration changes, and even switching the embedding model could cause regressions. The root cause wasn’t “no tests” but rather “no contract tests”—we hadn’t locked down the behavioral contract between the client and the vector database. Conventional mock tests can’t catch destructive changes in a real database, and end-to-end integration tests are too heavy and slow to run on every PR.

Solution Design

Tool selection: pytest + Allure for contract testing.

Why not other options?

• unittest: fixture design is not flexible enough, and the code becomes verbose.
• Postman/HTTP scripts: the Python SDK behavior is the core of the contract; HTTP can only test half of it.
• Pure mocks: a mocked query always returns what you expect, so it never tells you when the real SDK changes.
• Heavy integration tests: starting a real Docker service every time makes CI time explode, and developers don’t want to run them.

The architectural approach: run contract tests against Chroma’s EphemeralClient (in-memory mode) for speed and no external dependencies, while periodically running the same test suite against a real Docker service to verify consistency. Test cases cover four core contracts of memory storage: query after write, old content invisible after update, not queryable after delete, and count consistency. Allure generates readable reports so that non-developer colleagues can also understand regression results.

Core Implementation

1. Initialize the test environment and a custom embedding function

This code solves two problems: avoiding dependency on a real embedding model (slow and unstable), and providing each test function with an isolated in-memory collection.

# conftest.py
import chromadb
from chromadb import EmbeddingFunction, Documents, Embeddings
import pytest

class FakeEmbeddingFunction(EmbeddingFunction):
    """固定 8 维向量，避免依赖真实模型和网络"""
    def __call__(self, input: Documents) -> Embeddings:
        dim = 8
        vectors = []
        for text in input:
            vec = [0.0] * dim
            # 用字符编码生成确定性向量，保证同一文本永远映射到同一向量
            for i, ch in enumerate(text[:dim]):
                vec[i] = float(ord(ch) % 10) / 10.0
            vectors.append(vec)
        return vectors

@pytest.fixture(scope="function")
def memory_store():
    """每个测试用例独立的 Chroma 内存集合"""
    client = chromadb.EphemeralClient()
    embedding_fn = FakeEmbeddingFunction()
    collection = client.create_collection(
        name="memory_contract_test",
        embedding_function=embedding_fn,
        metadata={"hnsw:space": "cosine"}  # 使用余弦距离，和线上一致
    )
    yield client, collection
    # 清理：虽然 EphemeralClient 会自动销毁，但显式删除更保险
    try:
        client.delete_collection("memory_contract_test")
    except Exception:
        pass

2. Contract test cases: lock down the core behavior

This code directly verifies the four key contracts of memory storage, and every test uses allure decorators to output report steps.

# test_memory_contract.py
import allure

@allure.feature("向量数据库记忆存储契约")
class TestMemoryStoreContract:

    @allure.story("写入后查询")
    def test_upsert_and_query(self, memory_store):
        client, collection = memory_store
        with allure.step("写入一条记忆"):
            collection.upsert(
                ids=["mem-1"],
                documents=["用户对花生过敏"],
                metadatas=[{"user_id": "u1", "type": "allergy"}]
            )
        with allure.step("查询相似记忆"):
            results = collection.query(
                query_texts=["用户不能吃什么"],
                n_results=1,
                include=["documents", "metadatas", "distances"]
            )
        # 断言返回的 ID 和内容符合预期
        assert results["ids"][0] == ["mem-1"]
        assert "花生" in results["