The Vector Store Question That Changes Your Entire RAG Architecture

rahul shrivastava — Mon, 07 Sep 2026 06:35:25 +0000

I joined a RAG architecture review where the first question was:

“Which vector database should we use?”

Within minutes, the conversation became benchmark screenshots, vendor feature lists, GitHub stars, and pricing comparisons.

But nobody had asked the question that matters most:

“What retrieval problem are we actually solving?”

That is where many RAG projects make their first expensive mistake.

A vector store is not just where embeddings live. In production RAG, it determines whether the right evidence can be retrieved, filtered, secured, reranked, and returned within an acceptable latency target.

Here is the framework I use.

Start with retrieval shape

Are you searching 10,000 internal documents or 100 million chunks?

Do users ask broad questions, exact error-code questions, or both?

A small internal assistant may work well with Postgres and pgvector: familiar SQL, joins with business data, and less operational overhead.

A multi-tenant product with large collections, high query volume, fast ingestion, and strict latency targets may need a dedicated vector database or search platform.

The question is not, “Can it do vector search?”

Most options can.

The real question is, “Can it reliably support our workload at the expected scale?”

Make metadata filtering non-negotiable

Imagine an HR assistant retrieving a travel policy. It must be relevant, but also restricted by country, department, policy version, and employee permissions.

If filtering happens after retrieval, you may discard the right answer. If filters are slow, latency rises. If metadata is inconsistent, security risks grow.

Test a real query:

“Retrieve the current travel policy for an engineer in India, visible to this employee, from approved documents only.”

That reveals more than a generic nearest-neighbor benchmark.

Evaluate hybrid retrieval

Pure vector similarity is rarely enough.

Users search for error codes, ticket IDs, product SKUs, API fields, version numbers, and legal clauses. Keyword retrieval often wins for exact terms.

Production RAG commonly combines:

Dense vectors for semantic meaning

Keyword search for exact matches

Metadata filters for access control

Reranking for precision

Choose operational fit

Consider reindexing, monitoring, backup, recovery, tenant isolation, compliance, data residency, and cost as usage grows.

A store that looks excellent in a prototype can become difficult when you need to re-embed millions of chunks or investigate a failed retrieval.

My rule: choose the simplest vector store that meets your requirements for quality, filtering, scale, security, latency, and operations.

Start with real documents and 50–100 user questions. Define filters, evaluation metrics, latency targets, and failure cases. Then benchmark against your workload.

That turns a vector-store choice from a tooling debate into an engineering decision.

RAG #GenerativeAI #LLM #VectorDatabase #AIEngineering #SoftwareArchitecture #MLOps #TechLeadership #StaffEngineer #Hiring #AIJobs

Your RAG Is Only as Good as the Evidence Your Vector Index Retrieves

rahul shrivastava — Tue, 01 Sep 2026 17:38:40 +0000

Your RAG assistant may hallucinate—not because the LLM is weak, but because the vector index failed to retrieve the right evidence. 🔎

I saw this with a team whose RAG demo looked excellent at 50,000 chunks.

Queries were quick. Citations looked credible. Everyone was ready to scale.

Then the corpus reached millions of chunks.

p95 latency climbed. Retrieval became inconsistent. Important long-tail documents disappeared from the top results.

The first question was: “Should we switch the LLM?”

The real issue was lower in the stack: the vector index.

In RAG, an index is not merely a database setting. It decides how much of the corpus you can search within a latency and cost budget—and therefore affects grounded-answer quality.

Here is the trade-off:

Flat / brute force: compares against every vector. It gives exact nearest neighbours and is the right baseline for evaluation. But query cost grows linearly with the corpus.

IVF: groups vectors into clusters and searches selected clusters. It can be efficient and more memory-friendly, but may miss a relevant chunk outside the probed clusters. Raise nprobe for better recall at the cost of latency.

HNSW: connects similar vectors as a navigable graph. It commonly provides strong recall at low query latency, making it compelling for interactive RAG. The price: more RAM, longer builds, and extra operational tuning. Raise ef_search for better recall, again at a latency cost.

IVF + PQ/SQ8: compresses vectors to reduce memory and infrastructure cost. Useful at very large scale—but compression may hurt recall, especially for rare or highly specific queries.

For this team, a missed policy exception could create a confident but wrong answer.

So the decision was not “Which index is fastest?”

It was: “Which retrieval quality can this product afford to lose?”

The eventual pipeline was:

Hybrid retrieval → metadata/ACL filtering → ANN candidates → reranking → grounded answer with citations

The index determines candidate recall.
The reranker determines candidate order.
The LLM determines how well it uses evidence.

My rule of thumb:

Flat for truth and offline evaluation.
HNSW for high-quality, low-latency RAG when RAM is available.
IVF when build speed, memory, or cost constraints dominate.
Quantized IVF when full-precision vectors become impractical.

Do not select an index from a benchmark chart alone.

Measure Recall@K, p95/p99 latency, grounded-answer rate, citation correctness, real metadata-filter behaviour, ingestion lag, and cost per query.

The best RAG index is not the one that returns vectors fastest.

It is the one that retrieves the right evidence reliably enough to meet your product’s quality, latency, and cost promise.

What index are you using in production—and which metric drove that choice?

RAG #GenerativeAI #VectorSearch #AIEngineering

DEV Community: rahul shrivastava

The Vector Store Question That Changes Your Entire RAG Architecture

RAG #GenerativeAI #LLM #VectorDatabase #AIEngineering #SoftwareArchitecture #MLOps #TechLeadership #StaffEngineer #Hiring #AIJobs

Your RAG Is Only as Good as the Evidence Your Vector Index Retrieves

RAG #GenerativeAI #VectorSearch #AIEngineering