I joined a RAG architecture review where the first question was:
“Which vector database should we use?”
Within minutes, the conversation became benchmark screenshots, vendor feature lists, GitHub stars, and pricing comparisons.
But nobody had asked the question that matters most:
“What retrieval problem are we actually solving?”
That is where many RAG projects make their first expensive mistake.
A vector store is not just where embeddings live. In production RAG, it determines whether the right evidence can be retrieved, filtered, secured, reranked, and returned within an acceptable latency target.
Here is the framework I use.
- Start with retrieval shape
Are you searching 10,000 internal documents or 100 million chunks?
Do users ask broad questions, exact error-code questions, or both?
A small internal assistant may work well with Postgres and pgvector: familiar SQL, joins with business data, and less operational overhead.
A multi-tenant product with large collections, high query volume, fast ingestion, and strict latency targets may need a dedicated vector database or search platform.
The question is not, “Can it do vector search?”
Most options can.
The real question is, “Can it reliably support our workload at the expected scale?”
- Make metadata filtering non-negotiable
Imagine an HR assistant retrieving a travel policy. It must be relevant, but also restricted by country, department, policy version, and employee permissions.
If filtering happens after retrieval, you may discard the right answer. If filters are slow, latency rises. If metadata is inconsistent, security risks grow.
Test a real query:
“Retrieve the current travel policy for an engineer in India, visible to this employee, from approved documents only.”
That reveals more than a generic nearest-neighbor benchmark.
- Evaluate hybrid retrieval
Pure vector similarity is rarely enough.
Users search for error codes, ticket IDs, product SKUs, API fields, version numbers, and legal clauses. Keyword retrieval often wins for exact terms.
Production RAG commonly combines:
Dense vectors for semantic meaning
Keyword search for exact matches
Metadata filters for access control
Reranking for precision
- Choose operational fit
Consider reindexing, monitoring, backup, recovery, tenant isolation, compliance, data residency, and cost as usage grows.
A store that looks excellent in a prototype can become difficult when you need to re-embed millions of chunks or investigate a failed retrieval.
My rule: choose the simplest vector store that meets your requirements for quality, filtering, scale, security, latency, and operations.
Start with real documents and 50–100 user questions. Define filters, evaluation metrics, latency targets, and failure cases. Then benchmark against your workload.
That turns a vector-store choice from a tooling debate into an engineering decision.
Top comments (0)