Why mocking your Vector DB in CI is a terrible idea (and a 200ms fix)

Over the last year, building a RAG application has become incredibly easy. But if you’ve tried to move that prototype from a Jupyter Notebook to a production-grade CI/CD pipeline, you’ve probably hit a brick wall.

Testing vector search in a GitHub Actions runner without melting the server is a logistical nightmare. When setting up pytest for an app that relies on ChromaDB or Qdrant, engineering teams usually fall into one of two traps.

Trap 1: The Mocking Illusion

You decide to mock the database client using unittest.mock. Your test asserts that collection.query() was called with the right parameters.

Why it fails: You aren't actually testing anything useful. The entire point of a RAG pipeline is semantic search, the cosine similarity thresholds, embedding dimensions, and chunk retrieval. If you mock the vector database, you are flying blind. You’ll deploy to production only to realize your distance metrics were completely wrong.

Trap 2: The DinD Heavyweight

You decide to do it right and use Testcontainers to spin up a real ChromaDB instance in your CI workflow.

Why it fails: Running Docker-in-Docker (DinD) in a GitHub runner (which gives you 7GB of RAM) is painfully slow. Pulling gigabytes of vector database images and booting the daemon consumes massive amounts of memory. Your CI pipeline goes from 45 seconds to 10 minutes, or simply crashes with an OOMKilled error.

Separating Compute from State

The problem isn’t the database; it’s where we are running it. We are forcing our CI compute runners to also act as infrastructure provisioners.

The ideal architecture for testing is separating the two. Your CI runner should do nothing but execute your test code. The database should live on a remote, isolated server, boot instantly, and destroy itself the moment the test finishes.

To solve this, I built TrashDB. It’s an orchestration API (built in .NET 10 running on bare-metal) designed to do one thing: spin up databases in 200 milliseconds and kill them.

I recently shipped a native Python SDK so you can integrate it directly into your pytest fixtures without writing complex bash teardown scripts.

import pytest
import chromadb
from trashdb import TrashDBClient

@pytest.fixture(scope="session")
def vector_db():
    client = TrashDBClient()

    # Spins up an ephemeral ChromaDB container in ~200ms
    container = client.create_container(engine="chromadb", ttl_minutes=5)

    yield container

    # Cleans up the remote server automatically
    client.terminate_container(container.id)

def test_rag_retrieval(vector_db):
    chroma_client = chromadb.HttpClient(
        host=vector_db.host, 
        port=vector_db.port
    )

    # Your actual semantic search tests go here...

By externalizing the database, your runner only executes Python code. No heavy image pulls, no OOM crashes, and you can run multiple jobs in parallel without data collisions.

I’m building TrashDB in public and it's currently in a free Alpha. If you are struggling with testing your AI apps in CI, grab an API key at trashdb.dev.

Are you currently team mock, team Testcontainers, or something else entirely? Let me know below.

Comments

[Luis]

I've fallen into the "Mocking Illusion" trap before, and it's frustrating to realize that you're not actually testing the critical aspects of your RAG pipeline. The idea of separating compute from state and using an orchestration API like TrashDB to spin up ephemeral databases is really intriguing. I like how you've implemented the TrashDBClient as a fixture in your pytest example, making it easy to integrate into existing test workflows. Have you considered adding support for other vector databases like Qdrant or Pinecone, and how do you see TrashDB evolving to accommodate the growing landscape of AI and machine learning applications?

[Alvaro Lopez]

Hey! Thanks so much for reading and for the feedback.

To answer your questions:

1. Qdrant is actually already supported!
   You can spin it up exactly the same way. You just pass engine="qdrant" in the client and it provisions a remote instance in ~200ms, just like Chroma.

2. Regarding Pinecone and the future of TrashDB:
   Right now, I am strictly in the "idea validation" phase. I'm keeping the engine pool focused to see if developers actually adopt this "remote ephemeral DB" workflow for their CI/CD, or if they prefer to keep fighting with Docker and local resources.

If the community finds this architecture valuable and adoption grows, the product will absolutely evolve. Expanding the engine pool to support Pinecone, Weaviate, Milvus, and other ML-specific tools would be the natural next step, along with scaling the bare-metal infrastructure to support heavier loads.

If you end up trying the pytest fixture, let me know how it goes!