DEV Community: Rishabh Jain

Your RAG Agent Is Retrieving the Wrong Chunk: 5 Failure Modes We Fix in Production

Rishabh Jain — Tue, 16 Jun 2026 09:01:22 +0000

A client called us last month with a simple complaint: "Our support agent confidently quotes the wrong refund policy." The model was fine. The prompt was fine. The problem was three layers down, in the part nobody demos: retrieval. The agent was pulling the wrong chunk of text and then reasoning beautifully over the wrong facts.

This is the quiet truth about Retrieval-Augmented Generation (RAG). When an agent gives a wrong answer, the instinct is to blame the model or "prompt it harder." But in production, the majority of bad answers we debug are retrieval failures, not generation failures. The model did exactly what it was told - it just got handed the wrong context. Here are the five failure modes we see most often, and how we fix them.

1. Chunking that splits a fact in half

The default move is to slice documents into fixed 500-token windows. That works until a fact straddles a boundary - the eligibility rule is in chunk 14, the exception that voids it is in chunk 15, and your retriever returns only chunk 14. The agent now states a rule with total confidence and zero awareness of the exception.

The fix: chunk on structure, not character count. Split on headings, table rows, clauses, and list items. Add a small overlap (10-15%) so a fact and its caveat never get cleanly severed. For policy and contract data, we often store the whole section as one chunk even if it is long - a slightly bloated context beats an amputated fact.

2. Embeddings that confuse "similar words" with "same meaning"

Vector search retrieves what is semantically near the question. But "Can I cancel my subscription?" and "Can I cancel my appointment?" live close together in embedding space while meaning entirely different things in your system. Pure semantic search will happily hand back the appointment policy.

The fix: hybrid retrieval. Combine dense vector search with old-fashioned keyword (BM25) search and merge the results. Keywords catch the exact terms - product names, error codes, SKUs - that embeddings smudge together. In our experience this single change removes a large share of "close but wrong" retrievals.

3. No re-ranking, so the best chunk sits at position seven

Your retriever returns the top 20 candidates. The genuinely correct chunk is in there - at rank 7. But you only pass the top 3 to the model, so it never sees it. Recall was fine; ranking failed.

The fix: add a re-ranker. Pull a generous candidate set (say 20-30), then run a cross-encoder re-ranker that scores each chunk against the actual question and reorders them. Pass the top few after re-ranking. It is one extra step and it consistently lifts answer quality more than swapping to a bigger LLM.

4. Stale or duplicated documents

The 2023 pricing PDF and the 2026 pricing PDF both live in the index. Retrieval finds the 2023 one because it happens to be a tighter semantic match. Now your agent quotes prices from three years ago, and it is not wrong about the document - it is wrong about which document.

The fix: treat the index as a living dataset, not a one-time dump. Attach metadata (effective date, version, source) and filter on it at query time. Run a de-duplication pass. Re-index on a schedule. The most expensive RAG bugs we have untangled were not algorithmic - they were a forgotten stale file nobody removed.

5. No "I don't know" path

If retrieval returns nothing relevant, a naive pipeline still stuffs whatever it found into the prompt, and the model dutifully invents an answer. That is the hallucination everyone fears - except it was avoidable.

The fix: score the retrieval. If the top result's relevance is below a threshold, do not answer from it - say you do not have that information, or hand off to a human. An agent that knows the edge of its own knowledge is worth far more than one that bluffs.

How we test this before it ships

You cannot eyeball your way to a reliable RAG system. We build a small evaluation set - 50 to 100 real questions with known-correct source documents - and measure two things separately: retrieval accuracy (did we fetch the right chunk?) and answer accuracy (was the final response correct?). Splitting them tells you where the failure actually lives. Nine times out of ten, fixing retrieval fixes the answer, and you never needed a more expensive model at all.

RAG is not magic and it is not plumbing you can ignore. It is the layer that decides whether your agent is grounded in your business or improvising. Get it right and a mid-sized model outperforms a frontier model running on bad context. Get it wrong and no model on earth will save you.

About Shanti Infosoft: Shanti Infosoft is a CMMI Level 5 AI development company that has delivered 700+ projects across 16+ industries. We help teams move from AI ideas to dependable, production-grade software - shantiinfosoft.com | machine learning development services.

If your agent is confidently retrieving the wrong context, we can audit your retrieval pipeline and tune it against your own documents. Talk to our team.

Rishabh Jain is a Director at Shanti Infosoft, where the team builds AI agents and automation for real business operations.

Your RAG Agent Is Retrieving the Wrong Chunk: 5 Failure Modes We Fix in Production

Rishabh Jain — Tue, 16 Jun 2026 08:02:22 +0000

1. Chunking that splits a fact in half

2. Embeddings that confuse "similar words" with "same meaning"

3. No re-ranking, so the best chunk sits at position seven

Your retriever returns the top 20 candidates. The genuinely correct chunk is in there - at rank 7. But you only pass the top 3 to the model, so it never sees it. Recall was fine; ranking failed.

4. Stale or duplicated documents

5. No "I don't know" path

How we test this before it ships

If your agent is confidently retrieving the wrong context, we can audit your retrieval pipeline and tune it against your own documents. Talk to our team.

Rishabh Jain is a Director at Shanti Infosoft, where the team builds AI agents and automation for real business operations.

The Pilot-to-Production Gap: Why Your AI Agent Stalls After the Demo

Rishabh Jain — Tue, 16 Jun 2026 08:01:21 +0000

The agent looked unstoppable in the demo. Six weeks later it is still "almost ready to go live." If that sounds familiar, you are not alone, and the reason is almost never the model.

At Shanti Infosoft we have now built AI agents for support, sales, finance and operations teams, and the same pattern keeps repeating. The pilot dazzles everyone in a 20-minute meeting. Then it stalls in the gap between "it worked once on a clean example" and "it runs every day on the messy real thing." That gap is where most agent budgets quietly die. Crossing it is less about smarter AI and more about three unglamorous things: scope, ownership and operational readiness.

A demo proves possibility. Production demands reliability.

A demo is allowed to fail gracefully. You pick a good example, you narrate around the rough edges, everyone nods. Production is the opposite. It has to handle the weird ticket, the half-filled form, the customer who replies in three languages, the day your CRM is slow. The jump from "works on the happy path" to "survives the long tail" is the single biggest source of stall, and it is invisible in the demo precisely because the demo avoids it.

The fix is to stop demoing best cases. Before you celebrate a pilot, feed it a week of real, ugly historical data and watch where it breaks. The agent that handles your worst 20 percent of inputs is the one worth deploying. The one that only handles your best 20 percent is a slideshow.

Scope creep is the silent killer

The second stall is ambition. A pilot ships to do one thing. Then someone says "while we are at it, could it also..." and the agent grows three new responsibilities before the first one is trusted. Now nothing is reliable enough to launch, because every new branch adds new failure modes.

The teams that cross the gap do the opposite. They cut scope ruthlessly to get one workflow into real use, even a small one, and they let it earn trust before expanding. A narrow agent that reliably drafts first-pass support replies is in production. A grand "autonomous operations assistant" is in a backlog. Shipping the narrow one is not settling; it is how you build the track record that funds the bigger version.

Nobody budgeted for the boring 80 percent

Here is the line item that surprises clients most. Getting an agent to a working demo is maybe 20 percent of the effort. The other 80 percent is the unglamorous production work: connecting it safely to your real systems, handling errors, adding logging and audit trails, setting permissions, monitoring, and the inevitable tuning once real users touch it. If the project was scoped and funded as if the demo was the finish line, it runs out of money and energy exactly at the gap.

This is a planning problem, not a technology problem. Budget the production work as the main event, not the cleanup. A pilot that took two weeks to impress can easily take six to eight more to make dependable, and that is normal, not a failure.

No owner, no production

The quietest reason agents stall is that after the build, no single person owns the result. The vendor or internal builder hands it over, and it lands in a no-mans-land between IT, operations and the team that actually uses it. When something drifts, everyone assumes someone else is watching. So it sits at 90 percent forever.

Before you start, name one accountable owner who will live with the agent in production, decide when it is good enough to widen, and answer for it when it misbehaves. An agent with a named owner crosses the gap. An orphan does not.

How to actually cross it

If a pilot of yours is stuck, four moves usually unstick it. Test it on your worst real inputs, not your best. Cut the scope until one workflow can genuinely go live. Re-budget for the production 80 percent instead of treating it as polish. And give it a single accountable owner before launch, not after.

None of this requires a better model. It requires treating production as a deliberate phase with its own plan, not as the thing that happens automatically after a good demo.

The pilot-to-production gap is real, but it is crossable, and the teams that cross it are not the ones with the fanciest AI. They are the ones who planned for the boring part.

If you have an agent stuck at "almost ready," we are happy to take a look at where the gap actually sits in your setup. That diagnosis is often the cheapest part of getting unstuck.

If a pilot of yours has stalled before production, our team can help you find the real bottleneck and plan the work to get it live. Talk to our team.

Related reading: Your AI Demo Works. That's the Problem

Sagar Jain is a Director at Shanti Infosoft, where the team builds AI agents and automation for real business operations.

AI Agents for Business: A Practical Starting Point for 2026

Rishabh Jain — Mon, 15 Jun 2026 10:12:11 +0000

Every week another tool promises to "automate your business with AI." For most small and mid-sized teams, the hard part was never the model - it is figuring out which problems are actually worth handing to an AI agent, and which are better left alone.

At Shanti Infosoft we build AI agents and automation for real operations, not demos. Here is the practical starting point we share with clients in 2026.

Start with a painful, repetitive workflow

The best first agent is boring on purpose. Look for a task that runs many times a day, follows a fairly stable set of rules, and quietly eats your team's time: triaging support tickets, drafting first-pass replies, reconciling invoices, qualifying inbound leads, or summarising long documents. If a process is high-volume and rule-heavy, an agent will earn its keep fast. If it is rare or needs deep judgement, automate the edges and keep a human in the loop.

Keep a human in the loop at first

A useful agent does not have to be fully autonomous on day one. The fastest wins come from "draft and approve" patterns: the agent prepares the reply, the report, or the action, and a person clicks approve. You capture most of the time savings while keeping control - and the approvals become training data that tells you when the agent is ready for more autonomy.

Connect it to your real systems

An agent that cannot touch your CRM, inbox, database, or ticketing tool is just a chatbot. The value shows up when it reads and writes the systems your team already uses. This is where most DIY attempts stall, and where careful integration, permissions, and audit logging matter more than the model itself.

Measure before you scale

Pick one number before you launch - hours saved per week, response time, tickets deflected - and watch it for two weeks. A small agent that reliably saves five hours a week beats an ambitious one nobody trusts. Once the metric holds, widen the scope deliberately.

The takeaway

AI agents are not a single big bet. They are a series of small, well-scoped automations that compound. Start with one painful workflow, keep a human in the loop, wire it into your real systems, and measure the result. That is how automation actually sticks.

If you want a second opinion on where an agent would pay off in your business, that is exactly the kind of thing we help with at Shanti Infosoft.