<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Devanshu Patil</title>
    <description>The latest articles on DEV Community by Devanshu Patil (@ddp0411).</description>
    <link>https://dev.to/ddp0411</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1513318%2F11859fda-9d87-4399-bb80-ec9f48bd1097.jpg</url>
      <title>DEV Community: Devanshu Patil</title>
      <link>https://dev.to/ddp0411</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ddp0411"/>
    <language>en</language>
    <item>
      <title>Your RAG system is probably failing at retrieval, not generation</title>
      <dc:creator>Devanshu Patil</dc:creator>
      <pubDate>Tue, 17 Nov 2026 06:27:03 +0000</pubDate>
      <link>https://dev.to/ddp0411/your-rag-system-is-probably-failing-at-retrieval-not-generation-2pcb</link>
      <guid>https://dev.to/ddp0411/your-rag-system-is-probably-failing-at-retrieval-not-generation-2pcb</guid>
      <description>&lt;p&gt;There's a specific way RAG systems fail that I keep running into, in my own work&lt;br&gt;
and in other people's: &lt;strong&gt;retrieval quality degrades, and nothing on the dashboard&lt;br&gt;
moves.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The mechanism is simple once you see it. A language model handed incomplete&lt;br&gt;
context still writes a fluent, well-structured, confident answer. It reads as&lt;br&gt;
grounded. If your evaluation only scores the final answer, that answer looks&lt;br&gt;
fine — so a retriever that has quietly stopped surfacing the right passages&lt;br&gt;
generates no alarm at all.&lt;/p&gt;

&lt;p&gt;I learned this the hard way on &lt;a href="https://www.devanshupatil.blog/projects/neurorag" rel="noopener noreferrer"&gt;NeuroRAG&lt;/a&gt;, where for weeks my&lt;br&gt;
entire evaluation process was reading outputs and deciding whether they felt&lt;br&gt;
correct.&lt;/p&gt;

&lt;h2&gt;
  
  
  RAG has two stages and they fail independently
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Stage&lt;/th&gt;
&lt;th&gt;Question it answers&lt;/th&gt;
&lt;th&gt;How it fails&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Retrieve&lt;/td&gt;
&lt;td&gt;Did the right chunks come back for this query?&lt;/td&gt;
&lt;td&gt;Missed relevant chunks (low recall); surfaced irrelevant ones (low precision)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Generate&lt;/td&gt;
&lt;td&gt;Did the model use those chunks to answer correctly?&lt;/td&gt;
&lt;td&gt;Fabricated claims; ignored context; answered a different question&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Because they fail independently, &lt;strong&gt;you need at least one metric per stage.&lt;/strong&gt;&lt;br&gt;
Track only generation and retrieval regressions hide behind fluent prose. Track&lt;br&gt;
only retrieval and you miss the model fabricating on top of perfectly good&lt;br&gt;
context.&lt;/p&gt;

&lt;p&gt;That's the whole argument, and it's the thing I'd want someone to take away even&lt;br&gt;
if they read nothing else here.&lt;/p&gt;

&lt;h2&gt;
  
  
  The metrics worth having
&lt;/h2&gt;

&lt;p&gt;Scored on the retrieval span, independent of what the generator does with the&lt;br&gt;
results:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Context precision&lt;/strong&gt; — of the chunks retrieved, how many were actually
relevant? Low precision means you're burning context budget on noise.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Context recall&lt;/strong&gt; — of the chunks that &lt;em&gt;should&lt;/em&gt; have been retrieved, how many
were? This is the one that silently drifts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Precision@k / Recall@k, MRR, NDCG&lt;/strong&gt; — the classic IR metrics. MRR is
particularly useful when you care whether the single best passage came back
near the top.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Scored on the generation span:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Faithfulness / groundedness&lt;/strong&gt; — is every claim in the answer supported by the
retrieved context? This catches fabrication specifically.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Answer relevance&lt;/strong&gt; — did it answer the question that was asked, rather than a
neighbouring one?&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;⚠️ &lt;strong&gt;The trap&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Faithfulness and answer relevance can both look healthy while context recall falls off a cliff. The model is grounding faithfully in the incomplete context it received — it's being honest about the wrong material.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Score them on the same trace
&lt;/h2&gt;

&lt;p&gt;The detail that makes this practical: instrument retrieval and generation as&lt;br&gt;
spans on a &lt;strong&gt;single trace per query&lt;/strong&gt;, and attach metrics to their respective&lt;br&gt;
spans. When a score drops you want to open one trace and see the query, the&lt;br&gt;
retrieved chunks, the assembled prompt, and the output — in order.&lt;/p&gt;

&lt;p&gt;Without that, "faithfulness dropped 8% this week" is a fact you can't act on. You&lt;br&gt;
can't tell whether the retriever changed, the corpus changed, the chunking&lt;br&gt;
changed, or the prompt changed. With per-span traces, you look at ten failing&lt;br&gt;
examples and usually know within minutes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Build the smallest harness that works
&lt;/h2&gt;

&lt;p&gt;The version of this advice that gets ignored is the one that starts with "adopt&lt;br&gt;
an evaluation platform." You don't need one to begin. You need:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;A fixed question set.&lt;/strong&gt; Thirty to fifty questions covering the range of what
users actually ask, including questions your corpus &lt;em&gt;can't&lt;/em&gt; answer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Known-good source passages&lt;/strong&gt; for each. This is the tedious part and there's
no shortcut — you read the corpus and mark the right answers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A script&lt;/strong&gt; that runs the pipeline and reports retrieval hit rate at k, plus
whatever generation check you can afford.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A rule that you run it before every change you'd describe as an
improvement.&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;That's an afternoon of work and it converts "I think this helped" into "this&lt;br&gt;
moved recall@5 from 0.62 to 0.71." Everything more sophisticated is an upgrade&lt;br&gt;
from there, not a prerequisite.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;💡 &lt;strong&gt;On LLM-as-judge&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Using a model to score faithfulness and relevance scales well and is now standard practice. It also inherits the judge model's blind spots. Spot-check the judge against human labels on a sample before you trust its trend line — an unaudited judge is a metric with unknown bias.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Those unanswerable questions matter more than you'd think
&lt;/h2&gt;

&lt;p&gt;Point 1 above deserves emphasis. Include questions your corpus genuinely cannot&lt;br&gt;
answer, and check that the system declines rather than improvising.&lt;/p&gt;

&lt;p&gt;This is the single most useful test I run. A pipeline that answers everything is&lt;br&gt;
not a pipeline with great recall — it's a pipeline that has never learned to say&lt;br&gt;
no. For anything in a clinical, legal, or financial setting, a confident wrong&lt;br&gt;
answer is meaningfully worse than a refusal, and you only find out whether you&lt;br&gt;
have that property by explicitly testing for it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where this is heading
&lt;/h2&gt;

&lt;p&gt;The framing that's taken hold over the past year — and I think it's the right one&lt;br&gt;
— is that retrieval is one part of a broader &lt;strong&gt;context engineering&lt;/strong&gt; problem:&lt;br&gt;
what goes into the model's window, in what order, compressed how, carried across&lt;br&gt;
how many turns. Prompt phrasing matters much less than which three chunks the&lt;br&gt;
retriever surfaced.&lt;/p&gt;

&lt;p&gt;Graph-augmented retrieval is the interesting frontier here. Instead of returning&lt;br&gt;
isolated chunks, you return connected subgraphs — entities plus their&lt;br&gt;
relationships — which addresses the exact problem I hit with ICD-10, where a&lt;br&gt;
diagnosis and its exclusion criteria are related but not adjacent.&lt;/p&gt;

&lt;p&gt;I haven't rebuilt NeuroRAG on that model yet. When I do, I'll have a harness to&lt;br&gt;
tell me whether it actually helped — which is more than I could say the first&lt;br&gt;
time around.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://www.devanshupatil.blog/blog/your-rag-system-is-failing-at-retrieval" rel="noopener noreferrer"&gt;devanshupatil.blog&lt;/a&gt;. I write about data engineering, RAG and AI there.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>rag</category>
      <category>ai</category>
      <category>llm</category>
      <category>machinelearning</category>
    </item>
  </channel>
</rss>
