A vector store has one job: store vectors and find the ones closest to a query vector.
That's it. The implementation details — how it indexes vectors, how it handles persistence, how it scales — vary enormously between options. Understanding those details is what lets you make an informed choice rather than just using whatever the tutorial used.
This article is about the vector store layer in my RAG pipeline — how Chroma works under the hood, why it was the right choice for a local development tool, and what a production migration looks like.
What a Vector Store Is
When you embed a chunk of text, you get a vector — a list of 384 floating-point numbers representing that chunk's semantic meaning. A vector store does three things with that vector:
- Stores it alongside the original text and metadata
- Indexes it so similarity search is fast (not a linear scan through every stored vector)
- Retrieves the k most similar vectors to a query vector using a similarity metric The similarity metric is usually cosine similarity or L2 (Euclidean) distance. Cosine similarity measures the angle between vectors — two vectors pointing in the same direction have cosine similarity of 1.0, regardless of magnitude. L2 measures the straight-line distance between vector endpoints. Both work; the choice affects which embedding models pair best with which stores.
Chroma uses L2 distance by default, which is why my query results show distance=0.2341 — smaller is better (more similar).
Why Chroma for Local Development
import chromadb
client = chromadb.PersistentClient(path="chroma_db/")
collection = client.get_or_create_collection(
name="documents",
embedding_function=embedding_fn
)
Three lines. A persistent vector store on disk. No Docker container, no cloud account, no configuration files, no infrastructure.
This is Chroma's core value proposition for local development: zero operational overhead. The PersistentClient creates a chroma_db/ directory in your project, stores everything there as SQLite and binary files, and loads it back on the next run. It's a database that requires no database administration.
For a development pipeline where the primary goal is learning the RAG architecture and validating query quality, operational simplicity is the right priority. Every minute spent on infrastructure is a minute not spent on understanding the retrieval mechanics.
The Store Interface
The store module in my pipeline exposes two functions:
def add_documents(documents: list[dict]) -> int:
"""
Embed and store a list of chunked documents.
Returns the number of chunks added.
"""
texts = [doc["text"] for doc in documents]
ids = [f"{doc['source']}_{doc['chunk_index']}" for doc in documents]
metadatas = [{"source": doc["source"]} for doc in documents]
collection.add(
documents=texts,
ids=ids,
metadatas=metadatas
)
return len(documents)
def query(question: str, top_k: int = 5) -> list[dict]:
"""
Find the top_k most similar chunks to the question.
Returns chunks with their text, source, and distance.
"""
results = collection.query(
query_texts=[question],
n_results=top_k
)
chunks = []
for i, text in enumerate(results["documents"][0]):
chunks.append({
"text": text,
"source": results["metadatas"][0][i]["source"],
"distance": results["distances"][0][i]
})
return chunks
The interface is deliberately narrow. The loader doesn't call Chroma directly. The pipeline doesn't call Chroma directly. Everything goes through add_documents and query — two functions with stable signatures that any backing store can implement.
This is the swap point the README documents. Replacing Chroma with Pinecone means rewriting rag/store.py while everything else stays the same. The loader still produces the same document dictionaries. The pipeline still calls query() and gets the same chunk format back.
How Chroma Indexes Vectors
Chroma uses HNSW (Hierarchical Navigable Small World) indexing — the same algorithm used by most production vector stores. Understanding it at a high level is useful for knowing when it might not be the right choice.
HNSW builds a multi-layer graph where each vector is a node connected to its nearest neighbors. At query time, it navigates the graph starting from an entry point, greedily moving toward the query vector, pruning branches that are clearly not relevant. This gives approximate nearest-neighbor search — not guaranteed to find the exact closest vector, but finding very close ones in time proportional to log(n) rather than n.
For most RAG use cases, approximate nearest-neighbor is fine. If the fifth-most-similar chunk is returned instead of the exact fifth-closest, the quality difference is negligible. The logarithmic scaling is what makes vector search practical at millions of vectors.
Chroma's HNSW implementation runs in-process, which is why it needs no separate server. For development, this is a feature. For production, it's a limitation — the index lives in the application process's memory, which means it can't be shared across multiple instances of your application.
The Document ID Strategy
Document IDs in my pipeline are {source}_{chunk_index}:
ids = [f"{doc['source']}_{doc['chunk_index']}" for doc in documents]
This serves two purposes. First, it enables idempotent ingestion — if you run python cli.py ingest data/ twice, Chroma's add operation will update existing documents rather than creating duplicates, because the ID already exists. Second, it makes debugging tractable — if you know a specific chunk behaved unexpectedly, its ID tells you exactly which file and which chunk within that file.
A production ID strategy would be more robust: a hash of the content rather than a path-based ID, so the same content ingested from different paths gets the same ID and doesn't create duplicates.
What Chroma Doesn't Do
Being honest about Chroma's limitations in production contexts:
No multi-process sharing. The PersistentClient is a local file-based store. Multiple processes or services can't share the same collection without filesystem conflicts. Production systems need a client-server architecture — Chroma has a server mode (chroma_server) but it's a different operational model.
No access control. Chroma has no concept of users, permissions, or row-level security. Every query retrieves from the full collection. A multi-tenant RAG system — where user A shouldn't see user B's documents — requires access control built outside Chroma, either at the application layer or by using separate collections per user.
Limited horizontal scaling. Chroma is designed for single-node deployment. It doesn't have built-in sharding or replication. For very large document sets — tens of millions of chunks — it hits practical limits.
No audit logging. There's no built-in record of which queries were run, what was retrieved, or when. For compliance-relevant applications, this needs to be added at the application layer.
The Production Alternatives
Pinecone is the most commonly recommended hosted option. Fully managed, scales to billions of vectors, built-in access control, REST API. The operational model is pay-per-query and pay-per-storage. The swap from Chroma to Pinecone is straightforward — both have Python clients with similar query interfaces.
pgvector is a PostgreSQL extension that adds vector similarity search to a standard Postgres database. For teams already running Postgres, this is compelling — no new infrastructure, familiar operational model, SQL-based access control, existing backup and monitoring tooling. The tradeoff is that vector search performance at very large scale isn't as optimised as purpose-built vector databases.
Weaviate offers a middle ground — open-source, self-hosted, with more features than Chroma (access control, multi-tenancy, hybrid search) but less operational complexity than a full managed service. Popular in enterprise AppSec contexts where data leaving the network is a concern.
Qdrant is another open-source option gaining traction for its performance characteristics and Rust implementation (fast, low memory footprint).
For a security-focused application where data must stay on-premise — which covers most enterprise security tooling — pgvector or a self-hosted Weaviate or Qdrant deployment are the realistic production options. Pinecone's fully managed model involves sending your data to a third party, which requires a vendor security review and data classification analysis.
The Migration Path From This Pipeline
If I were taking this pipeline to production, the vector store migration would follow this sequence:
Step 1 — Add the server mode. Switch from chromadb.PersistentClient to chromadb.HttpClient pointing at a Chroma server. The interface is identical; the backing store is now a separate process that can be shared.
Step 2 — Add per-collection access control. Create separate collections per user or per team. The query function takes a collection_name parameter and only searches the collections the user is authorised to access.
Step 3 — Add audit logging. Wrap the query function to log every call: timestamp, user, query text, chunks retrieved, similarity distances. This is the audit trail that compliance requires.
Step 4 — Migrate to production store. Swap the backing store to Pinecone or pgvector. Re-embed and re-ingest the corpus into the new store. Validate retrieval quality against a test question set before cutting over.
The clean interface design means steps 1-3 can happen inside rag/store.py without touching the loader or pipeline. Step 4 is a rewrite of the store module — but everything that calls it stays unchanged.
Full source at github.com/pgmpofu/rag-pipeline. The store implementation is in rag/store.py.
Final article in this series: securing a RAG pipeline — the threats I designed against, the ones I didn't, and what a production security review of this architecture would look like.
Top comments (0)