Production-Grade RAG Retrieval Strategies Explained

Retrieval-Augmented Generation, or RAG, is often introduced with a very simple diagram:

User question
    ↓
Retrieve relevant documents
    ↓
Send context to the LLM
    ↓
Generate an answer

Looks easy!

Unfortunately, the phrase “retrieve relevant documents” is doing a heroic amount of work in that diagram.

In a production system, retrieval needs to handle natural-language questions, error codes, product IDs, document permissions, outdated policies, duplicated chunks and sometimes data that is not even stored as text.

A powerful language model cannot compensate for consistently poor retrieval. If the wrong context reaches the model, the result may be a beautifully written wrong answer, which is impressive, but not especially useful.

Getting retrieval right comes down to a few deliberate choices. Here's the framework:

Let us look at the retrieval strategies in detail, that matter most in production.

1. Sparse retrieval: exact words still matter

Sparse retrieval is traditional keyword based search. BM25 is the most common production approach.

It works especially well when the query contains exact terms such as:

• Error codes
• Product identifiers
• API names
• Database columns
• Legal wording
• Acronyms

Strengths

Sparse retrieval is fast, explainable and reliable for exact terminology.

Limitation

It can miss relevant documents that use different wording.

2. Dense retrieval: search by meaning

Dense retrieval uses embedding models to convert queries and document chunks into vectors.

The system then retrieves vectors that are close to the query vector using a similarity measure such as cosine similarity or dot product.

Dense retrieval is useful for:

• Natural-language questions
• Paraphrases
• Conceptual searches
• Differently worded explanations

It can understand that:

Remove an employee's system access

is related to:

Staff offboarding and account revocation

Strengths

Dense retrieval understands semantic meaning beyond exact vocabulary.

Limitation

It may weaken short identifiers, codes or domain-specific tokens.

3. Hybrid retrieval: use both

Hybrid retrieval combines sparse and dense retrieval.

BM25 results
      +
Dense vector results
      ↓
Result fusion
      ↓
Combined candidate list

Sparse search contributes exact matching.

Dense search contributes semantic matching.

The result lists can be merged using techniques such as Reciprocal Rank Fusion, commonly called RRF.

Hybrid retrieval is a strong default for enterprise knowledge bases because enterprise content usually contains both:

• Natural-language explanations
• Exact technical identifiers

For example, a troubleshooting query may contain an error code along with a description of the issue. Sparse retrieval captures the code, while dense retrieval captures the meaning of the description.

That is why hybrid retrieval is commonly a safer production starting point than pure vector search.

4. Metadata filtering: retrieve from the correct scope

Relevance alone is not enough.

A document may be highly relevant but belong to:

• The wrong department
• The wrong customer
• An expired policy version
• Another geographic region
• A user-restricted repository

Metadata filtering narrows the search before or during retrieval.

Typical filters include:

1. tenant
2. department
3. region
4. document type
5. language
6. version
7. effective date
8. access permissions

For example:

Find approved security policies for Australia updated after March 2026

The system can extract filters such as:

region = Australia
status = approved
updated_after = March 2026

Metadata filtering improves relevance and protects sensitive data.

5. Reranking: improve the final order

The first retrieval stage is usually designed for recall.

Its job is to find a reasonably broad set of possible candidates.

Hybrid retrieval
    ↓
Top 30–100 candidates

A reranker then evaluates the query and each candidate more carefully.

Top candidates
    ↓
Cross-encoder or semantic reranker
    ↓
Best 5–10 chunks

Reranking often provides a significant improvement in precision because it examines the query and document together.

However, it adds latency and cost, so it is normally applied only to a limited candidate set.

6. Query enhancements: useful, but not mandatory

Sometimes the original query is vague, incomplete or complex.

Several supporting techniques can help.

Multi-query retrieval

Generate a few alternative versions of the query, retrieve for each one and merge the results.

This can improve recall when the user's vocabulary differs from the document vocabulary.

Query decomposition

Break a complex question into smaller sub-questions.

For example:

Which policy changed,
which classifications depend on it,
and which downstream tables are affected?

can be decomposed into three retrieval tasks.

HyDE

HyDE generates a hypothetical answer or relevant passage, embeds it and uses it for retrieval.

This can help when the user's query is very short or written differently from the documents.

These techniques are useful, but they should not be enabled everywhere by default. Every extra LLM call adds latency, cost and another opportunity for creativity where creativity may not be required.

Use them when evaluation shows a specific recall problem.

7. Parent-child retrieval and MMR

Retrieving the correct chunk is not the same as providing enough context.

Parent-child retrieval

The system indexes small child chunks for precise matching but returns a larger parent section.

Small chunk matched
      ↓
Parent section retrieved
      ↓
Complete context sent to the LLM

This is useful for policies, manuals and technical documentation where a small paragraph may depend on nearby definitions or exceptions.

Maximal Marginal Relevance

MMR balances relevance with diversity.

It helps prevent the final context from containing five slightly different copies of the same paragraph.

Use it when repeated or overlapping chunks are a problem. It is not a replacement for reranking.

8. When text retrieval is not enough

Not every question should be answered using document embeddings.

Structured retrieval

Use SQL or APIs for:

• Counts
• Transactions
• Current status
• Inventory
• Metrics
• Exact records

For example:

How many failed transactions occurred yesterday?

This is a SQL question, not an invitation for semantic search to improvise.

Graph retrieval

Use graph retrieval when relationships are central:

classification rule
    → column
    → table
    → downstream application

It is useful for lineage, dependencies, ownership and multi-hop questions.

Multimodal retrieval

Use multimodal retrieval when important information exists in:

• Diagrams
• Charts
• Screenshots
• Scanned documents
• Presentation slides
• Forms

Text extraction alone may lose critical visual information.

A practical production baseline

A strong general-purpose pipeline looks like this:

User query
    ↓
Query preparation
    ↓
Metadata and access-control filters
    ↓
BM25 + dense retrieval
    ↓
RRF or score fusion
    ↓
Reranking
    ↓
Parent-context expansion
    ↓
Final evidence
    ↓
LLM answer with citations

This does not mean every RAG system needs every component.

A simple FAQ system may work well with dense retrieval alone. A technical support system may need sparse heavy hybrid retrieval. A financial analytics assistant may route most questions to SQL.

The retrieval strategy should follow the data and query patterns, not whichever architecture diagram currently has the most arrows.
Start with a simple baseline, evaluate retrieval quality and add complexity only when the evidence shows that it is needed.
Thanks!