DEV Community

Cover image for Qdrant vs ChromaDB for Homelab RAG: Persistence, Memory, and CPU Load
Guatu
Guatu

Posted on Originally published at guatulabs.dev

Qdrant vs ChromaDB for Homelab RAG: Persistence, Memory, and CPU Load

A Chroma collection built with chromadb.Client() lives only in process memory. Restart the script and every embedding is gone. No error, no warning, and the next query returns nothing. This is the default in half the RAG tutorials online, and it's how many homelab agent memory setups begin.

That's not a knock on Chroma. Chroma was built to get you from zero to working retrieval in a notebook quickly, and it does that well. But a homelab RAG setup isn't a notebook. It's a service that has to survive reboots, node drains, container image upgrades, and you forgetting about it for three months. Any vector database can run a demo. The real question is which one you can keep running on a small node without it eating your RAM or losing your data.

I run Qdrant for my agent memory stack (the hybrid retrieval setup and the 6-layer memory architecture both sit on it), so I have a bias. I'll be upfront about where it comes from and where Chroma is the better call.

Why You'd Choose

You usually hit this decision at one of two points.

The first is the prototype-to-service jump. You have a Python script that chunks some docs, embeds them, and answers questions. It works. Now you want an agent, an MCP server, and a cron job to all hit the same index. An in-process library stops being enough, and you need something with a network API, real persistence, and concurrent access.

The second is the resource wall. Your index grew from a few thousand chunks to a few hundred thousand, and the LXC or K8s node it lives on is now swapping, or a CPU core is pinned for minutes after every bulk ingest. At that point the differences in how each database handles memory and indexing start to matter more than the API ergonomics.

On homelab hardware it comes down to three things:

  • Persistence: what's on disk, when it gets there, and what survives a container recreate.
  • Memory: how much RAM the index needs, and whether you can push parts of it to disk.
  • CPU: where the embedding and indexing work actually happens, and whether you can cap it.

Option A: ChromaDB

Chroma is an embedding database aimed at developer speed. You can run it embedded inside your Python process or as a standalone server. Since the 0.4 rewrite it has kept metadata and documents in SQLite and vectors in an HNSW index. The 1.x line moved the core to Rust, which helped performance and concurrency a lot.

Strengths

The API is the nicest of any vector DB I've used. You add documents, it embeds them, you query with text, you get text back:

import chromadb

# Persists to disk. chromadb.Client() would NOT.
client = chromadb.PersistentClient(path="/data/chroma")
col = client.get_or_create_collection("notes")

col.add(ids=["n1", "n2"], documents=["Longhorn replica rebuild notes", "Traefik host rules"])
print(col.query(query_texts=["storage replication"], n_results=1)["documents"])
Enter fullscreen mode Exit fullscreen mode

Five lines and you have semantic search. No schema, no vector size declaration, no embedding pipeline. For exploring whether RAG even helps your use case, that's hard to beat.

It also stores your documents next to the vectors by default, which is convenient. One system, one backup, one place to look.

Weaknesses

The persistence footgun. Client() vs PersistentClient(path=...) is a one-word difference with total data loss on one side. The server mode has its own version of this: the data directory inside the official Docker image changed between the 0.x and 1.x images. If you mount a volume at the old path on the new image, Chroma writes to the container's writable layer. That works perfectly until the container gets recreated, and then the data is gone. Check the image docs for the version you're pinning, and verify persistence by recreating the container before you load real data.

The hidden CPU cost. That five-line example ran an embedding model. Chroma's default embedding function downloads a small sentence-transformer (all-MiniLM-L6-v2) and runs it on CPU through ONNX Runtime, in whatever process called add(). On a laptop you don't notice. On a 2-vCPU LXC ingesting a wiki, the ingest job saturates the CPU and the "database" gets blamed for load that's really embedding inference. You can (and should) pass your own embeddings, but the default hides where the work happens.

Memory is mostly all-or-nothing. Chroma's HNSW index for a collection has to fit in RAM to be queried. There's no first-class equivalent of keeping full-precision vectors on disk while searching a quantized copy in memory. For a homelab with a few hundred thousand chunks, that's fine. When memory gets tight, you have fewer knobs to turn.

Fewer operational primitives. No built-in snapshot API, no collection aliases for zero-downtime reindexing, thinner filtering. You back it up by copying the persist directory with the server stopped (or at least quiesced), which is fine but manual.

When it shines

Single-user tools, notebooks, prototypes, and small embedded apps where the vector store lives and dies with the application. If your entire RAG system is one Python process and a few thousand documents, Chroma is less infrastructure to own.

Option B: Qdrant

Qdrant is a vector search engine written in Rust, run as a standalone server with REST and gRPC APIs. It stores data in segments backed by a write-ahead log, builds HNSW indexes per segment, and has a fairly deep set of knobs for trading RAM against disk.

Strengths

Persistence is boring, which is what you want. Every write hits the WAL before it's acknowledged. Storage lives at /qdrant/storage in the official image, and that path has been stable. There's a snapshot API per collection, which makes backups something you can script instead of something you do by stopping the service. (Just don't leave the snapshots on the same disk as the data.)

Memory is tunable per collection. This is the big one for small hardware. You can keep original vectors memory-mapped on disk, keep the HNSW graph on disk, and search against a quantized int8 copy held in RAM:

from qdrant_client import QdrantClient, models

qd = QdrantClient(url="http://10.0.0.20:6333")
qd.create_collection(
    "memory",
    vectors_config={
        "dense": models.VectorParams(
            size=1024,
            distance=models.Distance.COSINE,
            on_disk=True,  # originals are mmap'd, not held in RAM
        )
    },
    quantization_config=models.ScalarQuantization(
        scalar=models.ScalarQuantizationConfig(
            type=models.ScalarType.INT8,
            always_ram=True,  # 4x smaller copy stays hot for search
        )
    ),
    on_disk_payload=True,
)
Enter fullscreen mode Exit fullscreen mode

Rough math for 100k chunks at 1024 dimensions: raw float32 vectors are 100,000 × 1024 × 4 bytes ≈ 410 MB. Qdrant's own sizing guidance adds about 50% for index and overhead, so plan on ~600 MB if everything sits in RAM. With int8 quantization in RAM and originals on disk, the hot set drops to roughly 100 MB plus the graph. With a fast NVMe behind it, rescoring against the on-disk originals barely changes recall.

CPU is cappable. Embedding happens wherever you decide (an Ollama endpoint, a FastEmbed process, the GPU node), never silently inside the database. The work Qdrant does do (search, background segment optimization, index building) can be bounded:

# config.yaml (or the matching QDRANT__STORAGE__PERFORMANCE__* env vars)
storage:
  performance:
    max_search_threads: 2        # cap concurrent search work
    max_optimization_threads: 1  # background merges/index builds stay on one core
Enter fullscreen mode Exit fullscreen mode

Idle, Qdrant does close to nothing. After a bulk upsert the optimizer kicks in to merge segments and build HNSW, and that's where you'll see CPU spike. Capping optimization threads makes the spike last longer instead of making it taller, which on a shared node is the trade you want.

Operational primitives exist. Collection aliases, payload indexes for filtered search, sparse vectors for hybrid BM25-style retrieval in the same collection, snapshots, and named vectors so one point can hold a dense and a sparse representation.

Weaknesses

More concepts up front. You declare vector size and distance. You manage the embedding pipeline yourself. You need to understand segments and the optimizer at least loosely, or the post-ingest CPU spike will look like a bug.

Named vectors also cut both ways. Newer releases of mcp-server-qdrant name the vector after the embedding model instead of using the default unnamed vector. Point an upgraded client at a collection built by an older one and you get a 400 Bad Request about a vector name that doesn't exist. Nothing is corrupted. The fix is a migration, and Qdrant's scroll API makes that a short script:

from qdrant_client import QdrantClient, models

qd = QdrantClient(url="http://10.0.0.20:6333")
NAME = "fast-bge-small-en"  # whatever name the new client expects
qd.create_collection("memory_v2", vectors_config={
    NAME: models.VectorParams(size=384, distance=models.Distance.COSINE)})

offset = None
while True:
    pts, offset = qd.scroll("memory", limit=256, offset=offset,
                            with_payload=True, with_vectors=True)
    qd.upsert("memory_v2", points=[
        models.PointStruct(id=p.id, vector={NAME: p.vector}, payload=p.payload)
        for p in pts])
    if offset is None:
        break
Enter fullscreen mode Exit fullscreen mode

Then repoint an alias at memory_v2 and drop the old collection once you've verified counts. Re-embedding isn't needed because the vectors themselves didn't change, only their name.

When it shines

Anything that's a long-lived service: agent memory, an MCP retrieval server shared by several tools, a wiki index that gets re-embedded on a schedule. Also anything where RAM is the binding constraint, because quantization plus on-disk storage lets you run indexes that wouldn't fit otherwise.

The Pattern That Matters More Than the Database

Whichever you pick, the biggest lever on memory and CPU isn't the engine. It's what you put in it.

Chroma's convenience pushes you toward storing full chunk text in the vector store. Do that with Qdrant payloads too, and every chunk's text rides along with its vector, gets indexed for filtering if you're careless, and bloats snapshots. I keep the vector DB as a reference index only: vectors plus a chunk_id and a few filterable fields. The text lives in SQLite, which is very good at being a key-value store for strings.

import sqlite3
from qdrant_client import QdrantClient

qd = QdrantClient(url="http://10.0.0.20:6333")
db = sqlite3.connect("/data/chunks.db")

def retrieve(query_vec: list[float], k: int = 5) -> list[dict]:
    hits = qd.query_points("memory", query=query_vec, using="dense", limit=k,
                           with_payload=["chunk_id", "title"]).points
    ids = [h.payload["chunk_id"] for h in hits]
    marks = ",".join("?" * len(ids))
    text = dict(db.execute(f"SELECT id, text FROM chunks WHERE id IN ({marks})", ids))
    return [{"title": h.payload["title"], "score": h.score,
             "text": text.get(h.payload["chunk_id"], "")} for h in hits]
Enter fullscreen mode Exit fullscreen mode

Two side benefits. Your vector DB becomes disposable: if a client upgrade changes vector names or you swap embedding models, you rebuild the index from SQLite without touching the source of truth. And retrieval becomes a policy decision. The MCP tool wrapping this function decides how much text leaves SQLite. For sensitive collections, mine return titles, scores, and categories only, and a separate tool with tighter permissions fetches full text. An agent that's been prompt-injected can enumerate what exists but can't bulk-dump it through the search tool.

This pattern also forces you to own index maintenance, which you should be doing anyway. A rebuild script that compares SQLite against the collection (not just by count; counts lie) catches drift before your agent starts confidently answering from stale chunks.

Decision Framework

Criterion ChromaDB Qdrant
Time to first query Minutes, embeds for you Longer, you bring embeddings
Default persistence In-memory unless you choose PersistentClient On-disk with WAL, always
Container data path Changed between 0.x and 1.x images Stable /qdrant/storage
Backups Copy the directory, ideally quiesced Snapshot API per collection
RAM for 100k × 1024-dim ~600 MB, all in RAM ~100 MB hot with int8 + on-disk originals
Disk offload of vectors/index Limited on_disk vectors, HNSW, and payload
Where embedding CPU goes Client process by default (ONNX on CPU) Wherever you put it
CPU caps Few knobs Search and optimizer thread limits
Hybrid dense + sparse Not native in the same way Named dense + sparse vectors per point
Zero-downtime reindex Manual Collection aliases
Breaking-change risk Image path and API shifts across majors Client-side vector naming changes
Best fit Prototypes, single-process apps Long-lived services, shared agent memory

My Pick and Why

Qdrant, for anything I expect to still be running next quarter.

The deciding factor isn't speed. At homelab scale both are fast enough that query latency is noise next to the LLM call. It's that Qdrant's failure modes are loud and recoverable while Chroma's most common ones are quiet. A wrong vector name gives you a 400 and a migration script. A wrong client constructor or volume path gives you an empty collection after a restart and a vague feeling that the agent got dumber. I'd rather debug the 400.

The memory controls seal it on small hardware. Being able to keep a quantized index hot while full-precision vectors sit on NVMe is the difference between an index that fits on a modest LXC and one that needs its own VM. And because Qdrant never embeds anything for you, CPU load is always attributable: if the node is pegged, I know whether it's the optimizer or my embedding job, and I can cap the right one.

Chroma still earns a spot. When I'm testing whether a new document set even benefits from retrieval, or evaluating chunking strategies, I'll reach for an embedded Chroma collection because the setup cost is near zero and the data is throwaway by design. If your whole RAG system is one process on one machine, it's a reasonable permanent choice too. Use PersistentClient, pass your own embeddings, and test a container recreate before you trust it.

Some lessons that hold regardless of engine:

  • Keep text out of the vector store. SQLite (or Postgres) as source of truth, vector DB as a rebuildable index. It cuts RAM and snapshot size, and it turns client upgrades into reindexes instead of data recovery.
  • Recreate the container before loading real data. It's a ten-second test for the entire class of "it was persisting to the wrong place" problems.
  • Move embedding off the database node. Embedding inference is the heaviest CPU work in most homelab RAG setups, and it belongs on the GPU node or at least on a process you can schedule and limit.
  • Script the rebuild on day one. Some upgrade will eventually change a vector name, an embedding model, or a schema, and a tested rebuild path turns that into a routine job.

If you're designing agent memory or a retrieval layer for something closer to production than a homelab, this is the kind of architecture work I do through GuatuLabs. For the retrieval-theory side of the same choice, vector search vs activation-based recall covers what the database can't solve for you.

Top comments (0)