Beyond Basic RAG: Building Production-Grade Agentic Workflows with Hybrid Search and Custom Re-Ranking

By Mithilesh Kumar | AI Engineer & Systems Architect

Standard Retrieval-Augmented Generation (RAG) pipelines often hit a wall in production environments. While simple vector similarity search works well for basic Q&A demos, real-world enterprise applications demand multi-step reasoning, precise contextual retrieval, and dynamic decision-making.

When building scalable AI systems, relying solely on dense vector embeddings leads to missing exact-keyword matches, failing on complex multi-part queries, and introducing unnecessary LLM hallucinations.

In this guide, I will walk you through building a Production-Grade Agentic RAG Architecture using Hybrid Search (Dense + Sparse), Cross-Encoder Re-Ranking, and Stateful Agent Workflows.

---

1. The Bottlenecks of Naive RAG in Production

In a standard Naive RAG setup, the pipeline follows a rigid pattern: User Query -> Embedding -> Vector DB Lookup -> LLM Context Window.

This approach fails in production due to three critical bottlenecks:

1. Semantic Shift & Loss of Keywords: Vector embeddings capture semantic meaning but struggle with specific product IDs, technical code snippets, or proper nouns.
2. Top-K Irrelevance: Retrieving top-k documents purely based on cosine similarity often pulls in contextually adjacent but factually useless chunks.
3. Single-Shot Failure: Real-world queries require multi-step decomposition. A single retrieval step cannot handle queries that require comparing two distinct documents or routing to different data sources.

---

2. The Architectural Blueprint

To solve these challenges, we replace the linear pipeline with an Agentic Workflow supported by a dual-retrieval and re-ranking engine.

Key Architectural Components:

• Query Router (Agentic Layer): Dynamically analyzes incoming intent and decides whether to route the request to a vector store, a relational database, or a web search API.
• Hybrid Search Engine: Combines BM25 (Sparse Keyword Search) and Dense Vector Search (e.g., OpenAI / BGE Embeddings) using Reciprocal Rank Fusion (RRF).
• Cross-Encoder Re-Ranker: A specialized model (like bge-reranker-large) that evaluates the exact query-document pairs to assign a true relevance score before sending context to the LLM.

---

3. Implementation: Hybrid Search & Re-Ranking Code

Here is a modular Python implementation demonstrating how to combine hybrid retrieval with a re-ranking step:

python
import numpy as np
from typing import List, Dict
from sentence_transformers import CrossEncoder

class ProductionRAGPipeline:
    def __init__(self, reranker_model_name: str = 'BAAI/bge-reranker-large'):
        # Initialize Cross-Encoder for precision scoring
        self.reranker = CrossEncoder(reranker_model_name)

    def reciprocal_rank_fusion(self, dense_results: List[Dict], sparse_results: List[Dict], k: int = 60) -> List[Dict]:
        """Combines Dense and Sparse search results using Reciprocal Rank Fusion (RRF)."""
        rrf_scores = {}

        for rank, doc in enumerate(dense_results):
            doc_id = doc['id']
            if doc_id not in rrf_scores:
                rrf_scores[doc_id] = {'doc': doc, 'score': 0.0}
            rrf_scores[doc_id]['score'] += 1.0 / (k + rank + 1)

        for rank, doc in enumerate(sparse_results):
            doc_id = doc['id']
            if doc_id not in rrf_scores:
                rrf_scores[doc_id] = {'doc': doc, 'score': 0.0}
            rrf_scores[doc_id]['score'] += 1.0 / (k + rank + 1)

        reranked_docs = sorted(rrf_scores.values(), key=lambda x: x['score'], reverse=True)
        return [item['doc'] for item in reranked_docs]

    def rerank_context(self, query: str, retrieved_docs: List[Dict], top_n: int = 3) -> List[Dict]:
        """Applies Cross-Encoder re-ranking on fused retrieval results."""
        if not retrieved_docs:
            return []

        pairs = [[query, doc['text']] for doc in retrieved_docs]
        scores = self.reranker.predict(pairs)

        for idx, score in enumerate(scores):
            retrieved_docs[idx]['rerank_score'] = float(score)

        final_sorted = sorted(retrieved_docs, key=lambda x: x['rerank_score'], reverse=True)
        return final_sorted[:top_n]


4. Benchmarks & Production Lessons Learned
When deploying this agentic RAG system in production, several key trade-offs emerged:

Latency vs. Precision: Cross-encoders add approximately 50-150ms of latency per request. Mitigate this by passing only the top 15-20 RRF-fused documents to the re-ranker.

Context Compression: Stripping out useless metadata before feeding documents into the final prompt reduced token costs by nearly 38%.

Agent Loops: Always enforce a hard ceiling (e.g., maximum 3 reflection iterations) on agent decision loops to prevent infinite fallback execution.

Conclusion
Transitioning from Naive RAG to an Agentic Hybrid Pipeline is necessary for building reliable, production-ready AI applications. By combining sparse keyword matching, dense semantic search, cross-encoder re-ranking, and dynamic query routing, you create a system that is robust, accurate, and cost-effective.

About the Author
Mithilesh Kumar is an AI Engineer and Systems Architect specializing in Large Language Models (LLMs), Multi-Agent Workflows, and Production-Grade RAG Systems. He focuses on building deterministic, low-latency AI software and enterprise systems.

Portfolio: mithilesh-kumar-ai-engineer.netlify.app

LinkedIn: linkedin.com/in/mithileshkumar001

GitHub: github.com/mithxcode

X (Twitter): x.com/MITHILESH_7781