Your RAG Is Only as Good as the Evidence Your Vector Index Retrieves

rahul shrivastava — Tue, 01 Sep 2026 17:38:40 +0000

Your RAG assistant may hallucinate—not because the LLM is weak, but because the vector index failed to retrieve the right evidence. 🔎

I saw this with a team whose RAG demo looked excellent at 50,000 chunks.

Queries were quick. Citations looked credible. Everyone was ready to scale.

Then the corpus reached millions of chunks.

p95 latency climbed. Retrieval became inconsistent. Important long-tail documents disappeared from the top results.

The first question was: “Should we switch the LLM?”

The real issue was lower in the stack: the vector index.

In RAG, an index is not merely a database setting. It decides how much of the corpus you can search within a latency and cost budget—and therefore affects grounded-answer quality.

Here is the trade-off:

Flat / brute force: compares against every vector. It gives exact nearest neighbours and is the right baseline for evaluation. But query cost grows linearly with the corpus.

IVF: groups vectors into clusters and searches selected clusters. It can be efficient and more memory-friendly, but may miss a relevant chunk outside the probed clusters. Raise nprobe for better recall at the cost of latency.

HNSW: connects similar vectors as a navigable graph. It commonly provides strong recall at low query latency, making it compelling for interactive RAG. The price: more RAM, longer builds, and extra operational tuning. Raise ef_search for better recall, again at a latency cost.

IVF + PQ/SQ8: compresses vectors to reduce memory and infrastructure cost. Useful at very large scale—but compression may hurt recall, especially for rare or highly specific queries.

For this team, a missed policy exception could create a confident but wrong answer.

So the decision was not “Which index is fastest?”

It was: “Which retrieval quality can this product afford to lose?”

The eventual pipeline was:

Hybrid retrieval → metadata/ACL filtering → ANN candidates → reranking → grounded answer with citations

The index determines candidate recall.
The reranker determines candidate order.
The LLM determines how well it uses evidence.

My rule of thumb:

Flat for truth and offline evaluation.
HNSW for high-quality, low-latency RAG when RAM is available.
IVF when build speed, memory, or cost constraints dominate.
Quantized IVF when full-precision vectors become impractical.

Do not select an index from a benchmark chart alone.

Measure Recall@K, p95/p99 latency, grounded-answer rate, citation correctness, real metadata-filter behaviour, ingestion lag, and cost per query.

The best RAG index is not the one that returns vectors fastest.

It is the one that retrieves the right evidence reliably enough to meet your product’s quality, latency, and cost promise.

What index are you using in production—and which metric drove that choice?

RAG #GenerativeAI #VectorSearch #AIEngineering

DEV Community: rahul shrivastava

Your RAG Is Only as Good as the Evidence Your Vector Index Retrieves

RAG #GenerativeAI #VectorSearch #AIEngineering