Introduction
When designing a RAG (Retrieval-Augmented Generation) system, understanding what's happening inside vector search lets you tune the trade-offs between accuracy, speed, and cost yourself.
This article covers the fundamentals of vector search:
- ANN search (Approximate Nearest Neighbor) and HNSW (Hierarchical Navigable Small World)
- Similarity and distance metrics (cosine similarity, dot product, L2 distance)
- Metadata filtering
- BM25 (keyword search)
and then summarizes practical rules of thumb for choosing between them when designing a RAG system.
What Is ANN Search?
In vector search, you look for the vectors in a corpus that are closest to a query embedding. Exact nearest neighbor search (kNN), which computes distances against every vector via brute force, gets linearly slower as the dataset grows.
ANN (Approximate Nearest Neighbor) search trades a small amount of accuracy (recall) for the ability to return "close enough" vectors quickly, even at large scale.
How HNSW Works
HNSW (Hierarchical Navigable Small World) is one of the most widely used ANN algorithms. It treats vectors as nodes in a graph, connecting nearby vectors with edges. Its defining feature is a hierarchical structure: upper layers are sparse and used for long jumps, while lower layers are dense and used for fine-grained search.
Search starts at the top layer and descends one layer at a time as it gets closer to the query vector. Think of it like flying between continents by plane, taking a train into a city, then walking the last stretch on foot. This reaches the neighborhood of the target vector using far fewer distance calculations than brute force.
Accuracy and speed are tuned mainly through two parameters.
| Parameter | Role | Increasing it |
|---|---|---|
M |
Number of edges per node | Improves recall, but increases memory usage and build time |
ef_search |
Number of candidates kept during search | Improves recall, but slows down queries |
Similarity and Distance Metrics
Three common metrics measure how "close" two vectors are.
| Metric | Calculation | Meaning | Characteristics |
|---|---|---|---|
| Cosine similarity | Cosine of the angle between two vectors | -1 to 1, closer to 1 = more similar | Ignores vector magnitude, compares direction only |
| Dot product | Sum of the products of each component | Higher value = more similar | Magnitude also affects the result; equals cosine similarity for normalized vectors |
| L2 distance (Euclidean) | Straight-line distance between two points | Smaller value = more similar | Reflects the actual positional difference, not just direction |
Text embeddings usually encode meaning in the vector's direction, with magnitude carrying little meaning, so cosine similarity — or dot product on normalized vectors — is the common choice. L2 distance shows up more often for embeddings such as image features, where magnitude itself is meaningful.
Metadata Filtering
Vectors can carry metadata beyond the raw content itself — category, date, tags, access permissions, and so on. Metadata filtering means narrowing candidates by these conditions, either before or while running the vector search.
Pre-filtering narrows candidates to those matching the conditions first, then runs vector search within that subset. It's more accurate, but the narrowing step itself has a cost.
Post-filtering runs ANN search first to over-fetch neighbor candidates, then excludes ones that don't match the metadata conditions. It reuses the ANN graph structure as-is, so it's fast, but if the filter rejects a large fraction of candidates, you can end up with too few results.
Newer vector databases such as Pinecone, Weaviate, and Qdrant often implement "filter-aware ANN," which factors filter conditions into the graph traversal itself — getting the best of both approaches in many cases.
BM25
The "keyword search" side of hybrid search in RAG is, in most implementations, an algorithm called BM25 (Best Matching 25). It's a refinement of TF-IDF and is the default scoring function in many full-text search engines, including Elasticsearch. Its score is driven mainly by three factors.
- Term frequency (TF): The more often a query term appears in a document, the higher the score — but each additional occurrence contributes less (it saturates).
- Inverse document frequency (IDF): Common words that appear across most documents count for less; rare, specific terms and proper nouns count for more.
- Document length normalization: Scores are adjusted relative to the average document length, so longer documents don't win purely by having more words.
Where vector search captures semantic closeness, BM25 captures lexical, word-level matches. BM25 tends to win on queries where exact matches matter — model numbers, proper nouns — while vector search tends to win on paraphrased or ambiguous queries. Combining the two in hybrid search lets each compensate for the other's weaknesses.
Best Practices for RAG Design
Here's how to combine everything above when actually designing a RAG system.
Distance Metric: Follow Your Embedding Model
Rather than choosing a metric yourself, match whatever metric your embedding model assumes at training time. Major text embedding models from OpenAI, Cohere, and others are trained assuming cosine similarity (or dot product on normalized vectors), so the practical default is cosine similarity — or dot product on normalized embeddings if you want to shave off compute. L2 distance rarely shows up in text RAG.
ANN/HNSW Parameters: Prioritize Recall
In RAG, missing even one relevant neighbor means the LLM never sees information it needed. So prioritize recall over query speed: set ef_search relatively high, and over-fetch more candidates than your final top-k (e.g., fetch 20-50 candidates if you'll ultimately use 5) before narrowing down.
For corpora up to roughly tens of thousands of vectors, sticking with exact kNN instead of ANN is a perfectly reasonable choice. It eliminates any recall loss from approximation, and brute-force search is often fast enough at that scale. ANN becomes genuinely necessary somewhere in the hundreds of thousands to millions of vectors.
Metadata Filtering: Pre-filtering Is Mandatory for Security
Requirements like access control and tenant isolation — where a document must never leak into results it shouldn't — always call for pre-filtering (or filter-aware ANN). With post-filtering, if only a few permitted documents happen to land in the ANN candidate set, you can end up with empty results, or worse, a bug that leaks unauthorized documents.
Filters meant purely to improve precision — date ranges, document type, and so on — are often fine with post-filtering (over-fetch candidates, then narrow), which has the advantage of a simpler implementation.
Best Practices for the Overall Retrieval Pipeline
RAG retrieval layers tend to combine more than plain vector search alone.
- Hybrid search: Combine vector search (semantic similarity) with keyword search such as BM25, then merge both scores. This shores up exact-match cases — model numbers, proper nouns — that embeddings tend to struggle with.
- Reranking: Over-fetch candidates via ANN, then use a higher-precision reranker (e.g., a cross-encoder) to narrow down to the final top-k. This corrects for ANN's own approximation error downstream.
- Metadata filtering typically narrows scope and permissions before search, and is combined with reranking after search.
Summary
| Element | Rule of thumb |
|---|---|
| Distance metric | Follow your embedding model's assumption (cosine similarity, or dot product on normalized vectors, by default) |
| ANN parameters | Prioritize recall; consider exact kNN for small corpora |
| Metadata filtering | Pre-filtering is mandatory for access control; post-filtering is fine otherwise |
| Retrieval pipeline | Vector search + BM25 hybrid + reranking is the standard setup |
Vector search isn't just about fetching neighbors quickly with ANN. Designing the metric choice, filter strategy, and downstream reranking together is what meaningfully moves the needle on a RAG system's answer quality.
Top comments (0)