DEV Community: Mithilesh Kumar

The RAG Pipeline I Wouldn't Build the Same Way Twice

Mithilesh Kumar — Fri, 18 Sep 2026 16:01:00 +0000

Subtitle: A practical look at failure modes, query routing, hybrid retrieval, and knowing when agentic workflows are actually worth the complexity.

The Illusion of a Perfect Prototype

Every developer's first naive RAG system feels like magic. You chunk a few PDFs, pass them through an embedding model, save the vectors to a database, and hook up a top-k similarity search to an LLM. It takes 50 lines of Python, runs in a couple of seconds, and answers basic questions surprisingly well.

Then real users show up.

They ask questions with precise technical identifiers (error code 0x80070005), temporal constraints ("What changed in our deployment policy last month?"), or broad multi-part requirements ("Compare our feature set with our competitor's pricing tier").

Suddenly, top-k vector similarity breaks down completely:

Exact strings get ignored because the semantic embedding space flattens precise tokens into vague concepts.
Short, high-level queries pull irrelevant chunks that happened to share superficial tone rather than domain substance.
Multi-part queries return fragments of information while missing the broader context needed to synthesize a coherent response.

This article documents how my architectural thinking moved away from "vector search by default" toward a system built around query intent, multi-strategy retrieval, and selective complexity.

The First Failure Modes: Why Naive Retrieval Fails

A simple RAG architecture assumes a linear pipeline:

User Query -> Vector Search (Top-k) -> Context Assembly -> LLM Generation

This model assumes that semantic similarity is equivalent to relevance. In practice, they are two very different metrics.

1. The Token Precision Loss

Dense vector embeddings condense the meaning of a text segment into a fixed-dimensional space (e.g., 768 or 1536 dimensions). This works well for conceptual queries ("How do I reset my credentials?"), but fails for token-exact queries ("What is the threshold for MAX_RETRY_ATTEMPTS in config.v2?").

Because the vector space prioritizes overall semantic meaning, the specific token string MAX_RETRY_ATTEMPTS loses its distinctiveness.

2. The Chunk Size Dilemma

If your chunks are too small (e.g., 200 tokens), you preserve fine-grained facts, but you lose the broader context needed to make sense of them. If your chunks are too large (e.g., 2000 tokens), you preserve context, but you dilute the relevance signal and waste the LLM's context window on noise.

3. The Top-k Fallacy

Static top-k retrieval assumes that the ideal context size is constant. For a simple question, k=2 might be plenty. For a complex synthesis query, k=10 might still be insufficient. Retrieving a fixed number of chunks forces a trade-off between missing critical information and swamping the generation prompt with distraction.

Query Routing: Treating Queries Differently

The first major architectural shift was realizing that every incoming query does not deserve the same retrieval path. Treating all user input identically is an architectural flaw.

Instead of passing every string directly to an embedding model, the system first passes the query through a fast classification layer—a lightweight query router.

Classification Categories:

Factual / Keyword-Exact: Route to a lexical index (BM25 or full-text search engine).
Conceptual / Intent-Based: Route to a vector index via dense embedding models.
Complex / Multi-Attribute: Route to a hybrid path that executes both lexical and semantic searches in parallel.
Conversational / System Instructions: Bypass retrieval completely to save latency and vector database compute.

Hybrid Retrieval and Exact Matching

To balance conceptual matching with token precision, hybrid retrieval combines sparse keyword search (BM25) with dense vector search.

Sparse models track exact word frequencies and term rarity, while dense models capture broader intent. Combining them ensures that exact product IDs or error codes aren't missed, while conceptual queries still return contextually relevant documents.

Once both retrievers return candidate lists, their score distributions must be normalized. A simple, effective technique for combining these distinct score metrics is Reciprocal Rank Fusion (RRF):

RRF_Score(d) = Sum of (1 / (60 + Rank(d)))

Where 60 is a constant that prevents high-ranking outliers from disproportionately dominating the output.

Re-Ranking and Context Selection

Retrieval models prioritize recall—getting all potentially relevant chunks into a candidate list. Generative LLMs, on the other hand, require precision—receiving only the most useful context to formulate an answer.

Passing 30 hybrid-retrieved candidate chunks directly into an LLM causes the "Lost in the Middle" phenomenon: transformer models pay disproportionate attention to information placed at the very beginning or the very end of their prompt context, often ignoring facts buried in the middle.

The Role of Cross-Encoders

Bi-encoder models (standard vector embeddings) process the query and document chunks independently to create static vector representations. This is fast, but it prevents the query terms from interacting directly with the document terms.

Cross-encoder re-rankers process the query and document chunk together through transformer layers. This allows full cross-attention between every query token and every document token. While too computationally expensive to run against millions of database documents, running a cross-encoder against the top 20 or 30 retrieved candidates adds minimal latency while significantly sharpening relevance.

Engineering reliable RAG systems requires testing retrieval strategies against real-world query failure modes rather than relying solely on synthetic benchmarks.

The Temptation to Add Agents Everywhere

When a static retrieval pipeline fails on multi-step reasoning, it's tempting to immediately rewrite the system as a fully autonomous agentic loop using frameworks like LangGraph or AutoGen.

An agentic loop gives the LLM tool access (e.g., query generation, external search, reflection) and allows it to run in a loop until it decides it has enough context to answer the user.

However, adding agentic loops introduces significant engineering trade-offs:

Non-Deterministic Latency: A standard hybrid RAG pipeline might execute in 400ms to 800ms. An agentic loop that decides to re-query, critique its own output, and loop three times can easily spike to 5–10 seconds.
Infinite Loops and Drift: Without tight control flow, agents can enter tool-call loops or wander away from the original query intent.
Cost Escalation: Every reflection loop and intermediate decision step multiplies token consumption.

When is an Agentic Workflow Actually Justified?

An agentic approach is worth the complexity when:

The query requires multi-step dependency resolution (e.g., "Find the deployment logs for the service that failed after yesterday's DB migration").
The retrieval strategy depends on the outcome of a previous retrieval step.
Self-correction is strictly necessary to validate structured output against a pre-defined schema.

If a query can be answered by routing it to a structured hybrid search path, adding an agentic framework is unnecessary overhead.

A Practical Composite Architecture

A production-grade pipeline layout that balances latency, deterministic behavior, and retrieval precision follows this execution flow:

User Query enters the system.
Query Router classifies the intent (Direct Pass, Hybrid Pipeline, or Agentic Loop).
If Hybrid Pipeline, execute Dense Vector Search and Sparse Lexical Search (BM25) in parallel.
Merge results using Reciprocal Rank Fusion (RRF).
Pass top candidate chunks through a Cross-Encoder Re-Ranker.
Send filtered, high-precision context to the LLM for Final Generation.

Architectural Lessons Learned

If I were rebuilding a RAG architecture from scratch today, these core engineering principles would guide my design decisions:

Optimize your data ingestion before tweaking models. Bad chunking, missing metadata, and poor document cleaning cannot be fixed by a top-tier re-ranker or an expensive LLM.
Treat vector databases as one component of a broader retrieval system. Vector search is a powerful feature, but it is not a complete search infrastructure on its own.
Measure retrieval precision independently from generation quality. Evaluate your retrieval candidate lists using metrics like MRR (Mean Reciprocal Rank) and NDCG before rating the final generated text output.
Keep the default execution path deterministic. Use routing heuristics to send 80% of standard queries through a fast, deterministic hybrid retrieval path. Reserve non-deterministic agentic loops for complex edge cases.

Previous Articles in This Series

If you found this deep-dive useful, check out my earlier technical guides on building production AI systems:

About the Author

Mithilesh Kumar | AI Engineer & Full-Stack Developer

Mithilesh Kumar is an AI Engineer and Full-Stack Developer who specializes in building autonomous applications, Multi-Agent Systems, Agentic AI workflows, and Retrieval-Augmented Generation (RAG) pipelines. He is currently pursuing his Bachelor of Technology (B.Tech) in Computer Science & Engineering at Chandigarh Engineering College, Landran. He designs scalable architectures by blending AI-driven automation with modern web technologies, focusing on production reliability, retrieval precision, and practical engineering trade-offs.

Portfolio: mithilesh-kumar-ai-engineer.netlify.app
GitHub: github.com/mithxcode
LinkedIn: linkedin.com/in/mithileshkumar-ai

Beyond Basic RAG: Building Production-Grade Agentic Workflows with Hybrid Search and Custom Re-Ranking

Mithilesh Kumar — Tue, 11 Aug 2026 14:30:22 +0000

By Mithilesh Kumar | AI Engineer & Systems Architect

Standard Retrieval-Augmented Generation (RAG) pipelines often hit a wall in production environments. While simple vector similarity search works well for basic Q&A demos, real-world enterprise applications demand multi-step reasoning, precise contextual retrieval, and dynamic decision-making.

When building scalable AI systems, relying solely on dense vector embeddings leads to missing exact-keyword matches, failing on complex multi-part queries, and introducing unnecessary LLM hallucinations.

In this guide, I will walk you through building a Production-Grade Agentic RAG Architecture using Hybrid Search (Dense + Sparse), Cross-Encoder Re-Ranking, and Stateful Agent Workflows.

1. The Bottlenecks of Naive RAG in Production

In a standard Naive RAG setup, the pipeline follows a rigid pattern: User Query -> Embedding -> Vector DB Lookup -> LLM Context Window.

This approach fails in production due to three critical bottlenecks:

Semantic Shift & Loss of Keywords: Vector embeddings capture semantic meaning but struggle with specific product IDs, technical code snippets, or proper nouns.
Top-K Irrelevance: Retrieving top-k documents purely based on cosine similarity often pulls in contextually adjacent but factually useless chunks.
Single-Shot Failure: Real-world queries require multi-step decomposition. A single retrieval step cannot handle queries that require comparing two distinct documents or routing to different data sources.

2. The Architectural Blueprint

To solve these challenges, we replace the linear pipeline with an Agentic Workflow supported by a dual-retrieval and re-ranking engine.

Key Architectural Components:

Query Router (Agentic Layer): Dynamically analyzes incoming intent and decides whether to route the request to a vector store, a relational database, or a web search API.
Hybrid Search Engine: Combines BM25 (Sparse Keyword Search) and Dense Vector Search (e.g., OpenAI / BGE Embeddings) using Reciprocal Rank Fusion (RRF).
Cross-Encoder Re-Ranker: A specialized model (like bge-reranker-large) that evaluates the exact query-document pairs to assign a true relevance score before sending context to the LLM.

3. Implementation: Hybrid Search & Re-Ranking Code

Here is a modular Python implementation demonstrating how to combine hybrid retrieval with a re-ranking step:

import numpy as np
from typing import List, Dict
from sentence_transformers import CrossEncoder

class ProductionRAGPipeline:
    def __init__(self, reranker_model_name: str = 'BAAI/bge-reranker-large'):
        # Initialize Cross-Encoder for precision scoring
        self.reranker = CrossEncoder(reranker_model_name)

    def reciprocal_rank_fusion(self, dense_results: List[Dict], sparse_results: List[Dict], k: int = 60) -> List[Dict]:
        """Combines Dense and Sparse search results using Reciprocal Rank Fusion (RRF)."""
        rrf_scores = {}

        for rank, doc in enumerate(dense_results):
            doc_id = doc['id']
            if doc_id not in rrf_scores:
                rrf_scores[doc_id] = {'doc': doc, 'score': 0.0}
            rrf_scores[doc_id]['score'] += 1.0 / (k + rank + 1)

        for rank, doc in enumerate(sparse_results):
            doc_id = doc['id']
            if doc_id not in rrf_scores:
                rrf_scores[doc_id] = {'doc': doc, 'score': 0.0}
            rrf_scores[doc_id]['score'] += 1.0 / (k + rank + 1)

        reranked_docs = sorted(rrf_scores.values(), key=lambda x: x['score'], reverse=True)
        return [item['doc'] for item in reranked_docs]

    def rerank_context(self, query: str, retrieved_docs: List[Dict], top_n: int = 3) -> List[Dict]:
        """Applies Cross-Encoder re-ranking on fused retrieval results."""
        if not retrieved_docs:
            return []

        pairs = [[query, doc['text']] for doc in retrieved_docs]
        scores = self.reranker.predict(pairs)

        for idx, score in enumerate(scores):
            retrieved_docs[idx]['rerank_score'] = float(score)

        final_sorted = sorted(retrieved_docs, key=lambda x: x['rerank_score'], reverse=True)
        return final_sorted[:top_n]

4. Benchmarks & Production Lessons Learned

When deploying this agentic RAG system in production, several key trade-offs emerged:

Latency vs. Precision: Cross-encoders add approximately 50-150ms of latency per request. Mitigate this by passing only the top 15-20 RRF-fused documents to the re-ranker.
Context Compression: Stripping out useless metadata before feeding documents into the final prompt reduced token costs by nearly 38%.
Agent Loops: Always enforce a hard ceiling (e.g., maximum 3 reflection iterations) on agent decision loops to prevent infinite fallback execution.

Conclusion

Transitioning from Naive RAG to an Agentic Hybrid Pipeline is necessary for building reliable, production-ready AI applications. By combining sparse keyword matching, dense semantic search, cross-encoder re-ranking, and dynamic query routing, you create a system that is robust, accurate, and cost-effective.

About the Author

Mithilesh Kumar is an AI Engineer and Systems Architect specializing in Large Language Models (LLMs), Multi-Agent Workflows, and Production-Grade RAG Systems. He focuses on building deterministic, low-latency AI software and enterprise systems.

Portfolio: mithilesh-kumar-ai-engineer.netlify.app
LinkedIn: linkedin.com/in/mithileshkumar001
GitHub: github.com/mithxcode
X (Twitter): x.com/MITHILESH_7781

Why Basic RAG Fails in Production and How Adaptive Query Routing Fixes It

Mithilesh Kumar — Sat, 08 Aug 2026 20:01:15 +0000

Most developers build Retrieval-Augmented Generation (RAG) pipelines assuming every user query needs a vector search. In production, this naive approach fails in three distinct scenarios:

Simple Queries: "Hi", "Who created this bot?", or general knowledge queries don't need expensive vector database lookups.

Ambiguous Queries: Vague user questions lead to noisy retrieval, diluting the LLM's context window with irrelevant chunks.

Out-of-Domain Queries: When the vector DB contains no relevant documents, naive RAG forces the LLM to hallucinate an answer based on poor context.

In this guide, I’ll break down how to implement Adaptive RAG with Dynamic Query Routing using LangChain, Vector Stores (Pinecone/Chroma), and FastAPI.

What is Adaptive RAG?
Instead of routing every request directly to vector retrieval, Adaptive RAG acts as an intent-aware orchestrator:

Classify Intent: Determine whether the query needs internal vector docs, web search, or a direct response.

Retrieve & Grade: Fetch documents, then evaluate their relevance score before generating the answer.

Fallback Circuit: If document relevance is low, trigger fallback web search (e.g., Tavily API) or ask the user for clarification.

Step 1: Building a Structured Router with Pydantic
We enforce a strict JSON output schema using Pydantic to ensure our routing decision is 100% deterministic.

from pydantic import BaseModel, Field
from typing import Literal

class RouteQuery(BaseModel):
"""Route a user query to the most appropriate data source."""

datasource: Literal["vectorstore", "web_search", "direct_llm"] = Field(
    ...,
    description="Given a user question, choose whether to route it to vectorstore, web search, or direct LLM."
)
reasoning: str = Field(
    ..., description="Brief explanation for the routing decision."
)

Step 2: Query Classification Node
Using function calling / structured output capabilities of LLMs (like Google Gemini or OpenAI):

from langchain_core.prompts import ChatPromptTemplate
from langchain_google_genai import ChatGoogleGenerativeAI

llm = ChatGoogleGenerativeAI(model="gemini-1.5-flash", temperature=0)
structured_router = llm.with_structured_output(RouteQuery)

system_prompt = """You are an expert at routing user queries.

Use 'vectorstore' for questions related to internal technical documents, architecture, or codebase.
Use 'web_search' for recent events, live news, or external context.
Use 'direct_llm' for greetings, general conversational queries, or basic coding syntax. """

route_prompt = ChatPromptTemplate.from_messages([
("system", system_prompt),
("human", "{question}")
])

question_router = route_prompt | structured_router

Test the router

result = question_router.invoke({"question": "What is the API endpoint for CGC-NEXUS event registration?"})
print(f"Destination: {result.datasource} | Reason: {result.reasoning}")

Output: Destination: vectorstore | Reason: Query asks about specific internal project endpoints.

Step 3: Integrating into Async FastAPI Endpoint
Here is how to expose the adaptive pipeline via a FastAPI service:

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI(title="Adaptive RAG Engine")

class QueryRequest(BaseModel):
question: str

@app.post("/api/v1/query")
async def process_query(request: QueryRequest):
try:
# Step 1: Route Query
decision = await question_router.ainvoke({"question": request.question})

    # Step 2: Execute based on intent
    if decision.datasource == "direct_llm":
        response = await llm.ainvoke(request.question)
        return {"source": "direct_llm", "answer": response.content}

    elif decision.datasource == "vectorstore":
        # Perform vector store search & hallucination check
        return {"source": "vectorstore", "answer": "Retrieved from vector database."}

    else:
        # Fallback to Web Search
        return {"source": "web_search", "answer": "Retrieved from web search fallback."}

except Exception as e:
    raise HTTPException(status_code=500, detail=str(e))

Key Production Insights
Cost Optimization: Filtering out trivial questions before vector search reduces API calls and vector database read costs by up to 40%.

Zero Hallucination Loop: By running a fast grader node on retrieved chunks, you ensure irrelevant text never enters the final LLM prompt context.

Latency Reduction: Direct LLM calls bypass embedding generation and vector lookup entirely, responding in under 300ms.

Connect & Explore Code
Live Portfolio: https://mithilesh-kumar-ai-engineer.netlify.app/

GitHub Repository: https://github.com/mithxcode

LinkedIn: https://www.linkedin.com/in/mithileshkumar001

X (Twitter): https://x.com/MITHILESH_7781

How are you handling ambiguous queries in your RAG pipelines? Drop a comment below!

Building Scalable Multi-Agent Workflows & RAG Pipelines with LangGraph and FastAPI

Mithilesh Kumar — Fri, 07 Aug 2026 05:14:38 +0000

Building fully autonomous AI systems requires moving beyond simple linear prompts to robust, stateful agentic workflows. In this article, I share my core architecture for engineering Multi-Agent Systems and Retrieval-Augmented Generation (RAG) pipelines using LangGraph, FastAPI, and Python.

Why Agentic Workflows and LangGraph?

Traditional LLM applications often struggle with complex, multi-step execution paths. By leveraging LangGraph, we can model agent interactions as state machines (graphs), allowing:

Cyclic execution loops for iterative refinement.
State management across multiple tool calls and reasoning steps.
Human-in-the-loop integration for safety and oversight.

Core Architecture Stack

A modern agentic AI pipeline relies on a clean, scalable setup:

Orchestration Layer: LangGraph / LangChain for handling agent routines and condition-based routing.
Backend API Service: FastAPI for ultra-fast async request handling and stream processing.
Retrieval Layer (RAG): Vector databases paired with hybrid search algorithms for low-latency context retrieval.
Foundation Models: OpenAI & Google Gemini APIs.

Key Takeaways for AI Engineers

Deterministic Routing: Always keep edge transitions explicit to prevent agent loops.
Structured Outputs: Enforce Pydantic schemas on LLM outputs for reliable API responses.
Observability: Track token usage and execution steps using tools like LangSmith.

About the Author

Hi! I am Mithilesh Kumar, an AI Engineer specializing in Multi-Agent Systems, Agentic Workflows, and Modern Web Technologies.

🌐 Explore my portfolio and live projects: https://mithilesh-kumar-ai-engineer.netlify.app/