Rost

Posted on Dec 5 • Originally published at glukhov.org

Vector Stores for RAG Comparison

#llm #rag #python #ai

Choosing the right vector store can make or break your RAG application's performance, cost, and scalability. This comprehensive comparison covers the most popular options in 2024-2025.

What is a Vector Store and Why RAG Needs One

A vector store is a specialized database designed to store and query high-dimensional embedding vectors. In Retrieval Augmented Generation (RAG) systems, vector stores serve as the knowledge backbone—they enable semantic similarity search that powers contextually relevant document retrieval.

When you build a RAG pipeline, documents are converted to embeddings (dense numerical vectors) by models like OpenAI's text-embedding-3-small or open-source alternatives like BGE and E5. For state-of-the-art multilingual performance, Qwen3 embedding and reranker models offer excellent integration with Ollama for local deployment. For multilingual and multimodal applications, cross-modal embeddings can bridge different data types (text, images, audio) into unified representation spaces. These embeddings capture semantic meaning, allowing you to find documents by meaning rather than exact keyword matches.

The vector store handles:

Storage of millions to billions of vectors
Indexing for fast approximate nearest neighbor (ANN) search
Filtering by metadata to narrow search scope
CRUD operations for maintaining your knowledge base

After retrieving relevant documents, reranking with embedding models can further improve retrieval quality by re-scoring candidates using more sophisticated similarity measures.

Quick Comparison Table

Vector Store	Type	Best For	Hosting	License
Pinecone	Managed	Production, zero-ops	Cloud only	Proprietary
Chroma	Embedded/Server	Prototyping, simplicity	Self-hosted	Apache 2.0
Weaviate	Server	Hybrid search, GraphQL	Self-hosted/Cloud	BSD-3
Milvus	Server	Scale, enterprise	Self-hosted/Cloud	Apache 2.0
Qdrant	Server	Rich filtering, Rust perf	Self-hosted/Cloud	Apache 2.0
FAISS	Library	Embedded, research	In-memory	MIT
pgvector	Extension	Postgres integration	Self-hosted	PostgreSQL

Detailed Vector Store Breakdown

Pinecone — The Managed Leader

Pinecone is a fully managed vector database built specifically for machine learning applications.

from pinecone import Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")
index = pc.Index("my-rag-index")

# Upsert vectors
index.upsert(vectors=[
    {"id": "doc1", "values": embedding, "metadata": {"source": "wiki"}}
])

# Query with metadata filtering
results = index.query(
    vector=query_embedding,
    top_k=5,
    filter={"source": {"$eq": "wiki"}}
)

Pros:

Zero infrastructure management
Excellent documentation and SDK support
Serverless tier with pay-per-query pricing
Fast query latency (~50ms P99)

Cons:

Cloud-only (no self-hosting)
Costs scale with usage
Vendor lock-in concerns

Best for: Teams prioritizing speed-to-production and operational simplicity.

Chroma — The Developer Favorite

Chroma positions itself as the "AI-native open-source embedding database." It's beloved for its simplicity and seamless integration with LangChain and LlamaIndex.

import chromadb

client = chromadb.Client()
collection = client.create_collection("my-docs")

# Add documents with auto-embedding
collection.add(
    documents=["Doc content here", "Another doc"],
    metadatas=[{"source": "pdf"}, {"source": "web"}],
    ids=["doc1", "doc2"]
)

# Query
results = collection.query(
    query_texts=["semantic search query"],
    n_results=5
)

Pros:

Dead simple API
Built-in embedding support
Works embedded (in-memory) or client-server
First-class LangChain/LlamaIndex integration

Cons:

Limited scalability for very large datasets
Fewer enterprise features
Persistence can be tricky in embedded mode

Best for: Prototyping, small-to-medium projects, and Python-first teams.

Weaviate — Hybrid Search Champion

Weaviate combines vector search with keyword (BM25) search and offers a GraphQL API. It's excellent for scenarios where hybrid search improves retrieval quality.

import weaviate

client = weaviate.Client("http://localhost:8080")

# Create schema with vectorizer
client.schema.create_class({
    "class": "Document",
    "vectorizer": "text2vec-openai",
    "properties": [{"name": "content", "dataType": ["text"]}]
})

# Hybrid search (vector + keyword)
result = client.query.get("Document", ["content"]) \
    .with_hybrid(query="RAG architecture", alpha=0.5) \
    .with_limit(5) \
    .do()

Pros:

Native hybrid search (alpha parameter balances vector/keyword)
Built-in vectorization modules
GraphQL query language
Multi-tenancy support

Cons:

Higher operational complexity
Steeper learning curve
Resource-intensive

Best for: Production applications needing hybrid search and GraphQL APIs.

Milvus — Enterprise Scale

Milvus is designed for billion-scale vector similarity search. It's the go-to choice for enterprise deployments requiring massive scale.

from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType

connections.connect("default", host="localhost", port="19530")

# Define schema
fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True),
    FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=1536)
]
schema = CollectionSchema(fields)
collection = Collection("documents", schema)

# Insert and search
collection.insert([[1, 2, 3], [embedding1, embedding2, embedding3]])
collection.search(
    data=[query_embedding],
    anns_field="embedding",
    param={"metric_type": "COSINE", "params": {"nprobe": 10}},
    limit=5
)

Pros:

Proven at billion-vector scale
Multiple index types (IVF, HNSW, DiskANN)
GPU acceleration support
Active enterprise community (Zilliz Cloud)

Cons:

Complex deployment (requires etcd, MinIO)
Overkill for small projects
Steeper operational overhead

Best for: Large-scale enterprise deployments and teams with DevOps capacity.

Qdrant — Performance Meets Filtering

Qdrant is written in Rust, offering excellent performance and rich metadata filtering capabilities. It's increasingly popular for production RAG.

from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance, PointStruct

client = QdrantClient("localhost", port=6333)

# Create collection
client.create_collection(
    collection_name="documents",
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE)
)

# Upsert with rich payload
client.upsert(
    collection_name="documents",
    points=[
        PointStruct(id=1, vector=embedding, payload={"category": "tech", "date": "2024-01"})
    ]
)

# Search with complex filtering
client.search(
    collection_name="documents",
    query_vector=query_embedding,
    query_filter={"must": [{"key": "category", "match": {"value": "tech"}}]},
    limit=5
)

Pros:

Excellent query performance (Rust)
Rich filtering with nested conditions
Quantization for memory efficiency
Good balance of features and simplicity

Cons:

Smaller ecosystem than Pinecone/Weaviate
Cloud offering is newer

Best for: Teams needing high performance with complex filtering requirements.

FAISS — The Research Workhorse

FAISS (Facebook AI Similarity Search) is a library, not a database. It's the foundation many vector DBs build upon.

import faiss
import numpy as np

# Create index
dimension = 1536
index = faiss.IndexFlatIP(dimension)  # Inner product similarity

# Add vectors
vectors = np.array(embeddings).astype('float32')
index.add(vectors)

# Search
D, I = index.search(query_embedding.reshape(1, -1), k=5)

Pros:

Blazing fast in-memory search
Multiple index types (Flat, IVF, HNSW, PQ)
GPU support
No network overhead

Cons:

No persistence (need to save/load manually)
No metadata filtering
No CRUD (rebuild index for updates)
Single-node only

Best for: Research, prototyping, and scenarios where vectors fit in memory.

pgvector — PostgreSQL Native

pgvector adds vector similarity search to PostgreSQL. Use your existing Postgres infrastructure for vectors.

Can I use a traditional database like PostgreSQL for vector search? Absolutely—pgvector makes this possible and practical.

-- Enable extension
CREATE EXTENSION vector;

-- Create table with vector column
CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    content TEXT,
    embedding vector(1536)
);

-- Create HNSW index
CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops);

-- Similarity search
SELECT id, content, embedding <=> '[0.1, 0.2, ...]' AS distance
FROM documents
WHERE category = 'tech'
ORDER BY distance
LIMIT 5;

Pros:

Use existing PostgreSQL skills/infra
ACID transactions with vectors
Combine relational queries with vector search
No new database to operate

Cons:

Performance ceiling vs. specialized DBs
Limited to PostgreSQL ecosystem
Index building can be slow

Best for: Teams already on PostgreSQL who want vectors without new infrastructure.

Choosing the Right Vector Store

Decision Framework

Start with these questions:

What's your scale?
- < 100K vectors → Chroma, pgvector, FAISS
- 100K - 10M vectors → Qdrant, Weaviate, Pinecone
- > 10M vectors → Milvus, Pinecone, Qdrant
Self-hosted or managed?
- Managed → Pinecone, Zilliz (Milvus), Weaviate Cloud
- Self-hosted → Qdrant, Milvus, Chroma, Weaviate
Do you need hybrid search?
- Yes → Weaviate, Elasticsearch
- No → Any option works
What's your filtering complexity?
- Simple → Chroma, Pinecone
- Complex nested filters → Qdrant, Weaviate
What's the difference between FAISS and dedicated vector databases? If you need persistence, distributed search, or production features—choose a database. FAISS is ideal for embedded research scenarios.

Common RAG Architecture Patterns

For production systems, consider advanced RAG variants like LongRAG for extended contexts, Self-RAG with self-reflection capabilities, or GraphRAG using knowledge graphs for more sophisticated retrieval strategies.

Pattern 1: Simple RAG with Chroma

Documents → Embeddings → Chroma → LangChain → LLM

Best for MVPs and internal tools.

Pattern 2: Production RAG with Qdrant

Documents → Embeddings → Qdrant (self-hosted)
                           ↓
                      FastAPI → LLM

Best for cost-conscious production deployments.

Pattern 3: Enterprise RAG with Pinecone

Documents → Embeddings → Pinecone (managed)
                           ↓
                      Your App → LLM

Best for teams prioritizing reliability over cost.

When integrating LLMs into your RAG pipeline, structured output techniques with Ollama and Qwen3 can help ensure consistent, parseable responses from your language model, making it easier to extract and process retrieved information.

Performance Benchmarks

Real-world performance varies by dataset, queries, and hardware. General observations:

Operation	FAISS	Qdrant	Milvus	Pinecone	Chroma
Insert 1M vectors	30s	2min	3min	5min	4min
Query latency (P50)	1ms	5ms	10ms	30ms	15ms
Query latency (P99)	5ms	20ms	40ms	80ms	50ms
Memory/1M vectors	6GB	8GB	10GB	N/A	8GB

Note: Pinecone latency includes network overhead; others are local.

Migration Considerations

How do I choose between Chroma and Weaviate for my RAG project? Consider your migration path too:

Chroma → Production: Export embeddings, re-import to Qdrant/Pinecone
pgvector → Specialized: Use COPY to export, transform, and load
FAISS → Database: Save index, load vectors into target DB

Most frameworks (LangChain, LlamaIndex) abstract vector stores, making migration easier at the application layer.

Cost Comparison

Managed Options (monthly, 1M vectors, 10K queries/day):

Pinecone Serverless: ~$50-100
Pinecone Standard: ~$70-150
Weaviate Cloud: ~$25-100
Zilliz Cloud: ~$50-200

Self-Hosted (infrastructure cost):

Small VM (4GB RAM): $20-40/month
Medium VM (16GB RAM): $80-150/month
Kubernetes cluster: $200+/month

DEV Community

Vector Stores for RAG Comparison

What is a Vector Store and Why RAG Needs One

Quick Comparison Table

Detailed Vector Store Breakdown

Pinecone — The Managed Leader

Chroma — The Developer Favorite

Weaviate — Hybrid Search Champion

Milvus — Enterprise Scale

Qdrant — Performance Meets Filtering

FAISS — The Research Workhorse

pgvector — PostgreSQL Native

Choosing the Right Vector Store

Decision Framework

Common RAG Architecture Patterns

Performance Benchmarks

Migration Considerations

Cost Comparison

Useful Links

Top comments (0)