Subtitle: A practical look at failure modes, query routing, hybrid retrieval, and knowing when agentic workflows are actually worth the complexity.
The Illusion of a Perfect Prototype
Every developer's first naive RAG system feels like magic. You chunk a few PDFs, pass them through an embedding model, save the vectors to a database, and hook up a top-k similarity search to an LLM. It takes 50 lines of Python, runs in a couple of seconds, and answers basic questions surprisingly well.
Then real users show up.
They ask questions with precise technical identifiers (error code 0x80070005), temporal constraints ("What changed in our deployment policy last month?"), or broad multi-part requirements ("Compare our feature set with our competitor's pricing tier").
Suddenly, top-k vector similarity breaks down completely:
- Exact strings get ignored because the semantic embedding space flattens precise tokens into vague concepts.
- Short, high-level queries pull irrelevant chunks that happened to share superficial tone rather than domain substance.
- Multi-part queries return fragments of information while missing the broader context needed to synthesize a coherent response.
This article documents how my architectural thinking moved away from "vector search by default" toward a system built around query intent, multi-strategy retrieval, and selective complexity.
The First Failure Modes: Why Naive Retrieval Fails
A simple RAG architecture assumes a linear pipeline:
User Query -> Vector Search (Top-k) -> Context Assembly -> LLM Generation
This model assumes that semantic similarity is equivalent to relevance. In practice, they are two very different metrics.
1. The Token Precision Loss
Dense vector embeddings condense the meaning of a text segment into a fixed-dimensional space (e.g., 768 or 1536 dimensions). This works well for conceptual queries ("How do I reset my credentials?"), but fails for token-exact queries ("What is the threshold for MAX_RETRY_ATTEMPTS in config.v2?").
Because the vector space prioritizes overall semantic meaning, the specific token string MAX_RETRY_ATTEMPTS loses its distinctiveness.
2. The Chunk Size Dilemma
If your chunks are too small (e.g., 200 tokens), you preserve fine-grained facts, but you lose the broader context needed to make sense of them. If your chunks are too large (e.g., 2000 tokens), you preserve context, but you dilute the relevance signal and waste the LLM's context window on noise.
3. The Top-k Fallacy
Static top-k retrieval assumes that the ideal context size is constant. For a simple question, k=2 might be plenty. For a complex synthesis query, k=10 might still be insufficient. Retrieving a fixed number of chunks forces a trade-off between missing critical information and swamping the generation prompt with distraction.
Query Routing: Treating Queries Differently
The first major architectural shift was realizing that every incoming query does not deserve the same retrieval path. Treating all user input identically is an architectural flaw.
Instead of passing every string directly to an embedding model, the system first passes the query through a fast classification layer—a lightweight query router.
Classification Categories:
- Factual / Keyword-Exact: Route to a lexical index (BM25 or full-text search engine).
- Conceptual / Intent-Based: Route to a vector index via dense embedding models.
- Complex / Multi-Attribute: Route to a hybrid path that executes both lexical and semantic searches in parallel.
- Conversational / System Instructions: Bypass retrieval completely to save latency and vector database compute.
Hybrid Retrieval and Exact Matching
To balance conceptual matching with token precision, hybrid retrieval combines sparse keyword search (BM25) with dense vector search.
Sparse models track exact word frequencies and term rarity, while dense models capture broader intent. Combining them ensures that exact product IDs or error codes aren't missed, while conceptual queries still return contextually relevant documents.
Once both retrievers return candidate lists, their score distributions must be normalized. A simple, effective technique for combining these distinct score metrics is Reciprocal Rank Fusion (RRF):
RRF_Score(d) = Sum of (1 / (60 + Rank(d)))
Where 60 is a constant that prevents high-ranking outliers from disproportionately dominating the output.
Re-Ranking and Context Selection
Retrieval models prioritize recall—getting all potentially relevant chunks into a candidate list. Generative LLMs, on the other hand, require precision—receiving only the most useful context to formulate an answer.
Passing 30 hybrid-retrieved candidate chunks directly into an LLM causes the "Lost in the Middle" phenomenon: transformer models pay disproportionate attention to information placed at the very beginning or the very end of their prompt context, often ignoring facts buried in the middle.
The Role of Cross-Encoders
Bi-encoder models (standard vector embeddings) process the query and document chunks independently to create static vector representations. This is fast, but it prevents the query terms from interacting directly with the document terms.
Cross-encoder re-rankers process the query and document chunk together through transformer layers. This allows full cross-attention between every query token and every document token. While too computationally expensive to run against millions of database documents, running a cross-encoder against the top 20 or 30 retrieved candidates adds minimal latency while significantly sharpening relevance.
Engineering reliable RAG systems requires testing retrieval strategies against real-world query failure modes rather than relying solely on synthetic benchmarks.
The Temptation to Add Agents Everywhere
When a static retrieval pipeline fails on multi-step reasoning, it's tempting to immediately rewrite the system as a fully autonomous agentic loop using frameworks like LangGraph or AutoGen.
An agentic loop gives the LLM tool access (e.g., query generation, external search, reflection) and allows it to run in a loop until it decides it has enough context to answer the user.
However, adding agentic loops introduces significant engineering trade-offs:
- Non-Deterministic Latency: A standard hybrid RAG pipeline might execute in 400ms to 800ms. An agentic loop that decides to re-query, critique its own output, and loop three times can easily spike to 5–10 seconds.
- Infinite Loops and Drift: Without tight control flow, agents can enter tool-call loops or wander away from the original query intent.
- Cost Escalation: Every reflection loop and intermediate decision step multiplies token consumption.
When is an Agentic Workflow Actually Justified?
An agentic approach is worth the complexity when:
- The query requires multi-step dependency resolution (e.g., "Find the deployment logs for the service that failed after yesterday's DB migration").
- The retrieval strategy depends on the outcome of a previous retrieval step.
- Self-correction is strictly necessary to validate structured output against a pre-defined schema.
If a query can be answered by routing it to a structured hybrid search path, adding an agentic framework is unnecessary overhead.
A Practical Composite Architecture
A production-grade pipeline layout that balances latency, deterministic behavior, and retrieval precision follows this execution flow:
- User Query enters the system.
- Query Router classifies the intent (Direct Pass, Hybrid Pipeline, or Agentic Loop).
- If Hybrid Pipeline, execute Dense Vector Search and Sparse Lexical Search (BM25) in parallel.
- Merge results using Reciprocal Rank Fusion (RRF).
- Pass top candidate chunks through a Cross-Encoder Re-Ranker.
- Send filtered, high-precision context to the LLM for Final Generation.
Architectural Lessons Learned
If I were rebuilding a RAG architecture from scratch today, these core engineering principles would guide my design decisions:
- Optimize your data ingestion before tweaking models. Bad chunking, missing metadata, and poor document cleaning cannot be fixed by a top-tier re-ranker or an expensive LLM.
- Treat vector databases as one component of a broader retrieval system. Vector search is a powerful feature, but it is not a complete search infrastructure on its own.
- Measure retrieval precision independently from generation quality. Evaluate your retrieval candidate lists using metrics like MRR (Mean Reciprocal Rank) and NDCG before rating the final generated text output.
- Keep the default execution path deterministic. Use routing heuristics to send 80% of standard queries through a fast, deterministic hybrid retrieval path. Reserve non-deterministic agentic loops for complex edge cases.
Previous Articles in This Series
If you found this deep-dive useful, check out my earlier technical guides on building production AI systems:
- Building Scalable Multi-Agent Workflows & RAG Pipelines with LangGraph and FastAPI
- Why Basic RAG Fails in Production and How Adaptive Query Routing Fixes It
- Beyond Basic RAG: Building Production-Grade Agentic Workflows with Hybrid Search and Custom Re-Ranking
About the Author
Mithilesh Kumar | AI Engineer & Full-Stack Developer
Mithilesh Kumar is an AI Engineer and Full-Stack Developer who specializes in building autonomous applications, Multi-Agent Systems, Agentic AI workflows, and Retrieval-Augmented Generation (RAG) pipelines. He is currently pursuing his Bachelor of Technology (B.Tech) in Computer Science & Engineering at Chandigarh Engineering College, Landran. He designs scalable architectures by blending AI-driven automation with modern web technologies, focusing on production reliability, retrieval precision, and practical engineering trade-offs.
- Portfolio: mithilesh-kumar-ai-engineer.netlify.app
- GitHub: github.com/mithxcode
- LinkedIn: linkedin.com/in/mithileshkumar-ai
Top comments (1)
This separation between retrieval and generation is the part teams often skip. I’ve found route choice itself should be captured as evidence—lexical/vector/hybrid, the candidate set before reranking, and the score changes—so a bad answer can be traced to retrieval rather than blamed on the model. Have you found a compact per-query trace format that stays useful once agentic multi-step retrieval enters the pipeline?