<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Md Ayan Arshad</title>
    <description>The latest articles on DEV Community by Md Ayan Arshad (@ayanarshad02).</description>
    <link>https://dev.to/ayanarshad02</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3817591%2F0fa1e4d1-8be4-4467-b7f3-2fa3543e804b.png</url>
      <title>DEV Community: Md Ayan Arshad</title>
      <link>https://dev.to/ayanarshad02</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ayanarshad02"/>
    <language>en</language>
    <item>
      <title>One Agent Framework Seemed to Lose Half Its Citations. Most Were Just Reformatted.</title>
      <dc:creator>Md Ayan Arshad</dc:creator>
      <pubDate>Sun, 25 Oct 2026 16:12:37 +0000</pubDate>
      <link>https://dev.to/ayanarshad02/one-agent-framework-seemed-to-lose-half-its-citations-most-were-just-reformatted-2h1l</link>
      <guid>https://dev.to/ayanarshad02/one-agent-framework-seemed-to-lose-half-its-citations-most-were-just-reformatted-2h1l</guid>
      <description>&lt;p&gt;&lt;em&gt;Five pre-registered experiments, one at a time, the moment reading the raw answers changed the conclusion, the re-score, and the three runs per setup that finally told me what was actually true.&lt;/em&gt;&lt;/p&gt;




&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Part 2 of 6 in the series &lt;em&gt;Citations at the MCP Boundary&lt;/em&gt;.&lt;/strong&gt; New here? Start with &lt;a href="https://dev.to/ayanarshad02/your-mcp-server-returns-the-source-does-the-answer-still-cite-it-26n8"&gt;Part 1: Your MCP Server Returns the Source. Does the Answer Still Cite It?&lt;/a&gt;. The full list of parts is at the end of this post.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In &lt;a href="https://dev.to/ayanarshad02/your-mcp-server-returns-the-source-does-the-answer-still-cite-it-26n8"&gt;Part 1&lt;/a&gt; I described a benchmark for whether citations survive when an AI agent consumes a RAG knowledge base over MCP. On the first pass, the headline was uncomfortable for one framework:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Pipeline (gpt-4o-mini, same prompt, same 127 questions)&lt;/th&gt;
&lt;th&gt;CSR&lt;/th&gt;
&lt;th&gt;95% CI&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Single-shot RAG&lt;/td&gt;
&lt;td&gt;0.618&lt;/td&gt;
&lt;td&gt;[0.528, 0.708]&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LangGraph agent&lt;/td&gt;
&lt;td&gt;0.614&lt;/td&gt;
&lt;td&gt;[0.524, 0.705]&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenAI Agents SDK agent&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.323&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;[0.237, 0.414]&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A gap that size between two agents running the same model and the same prompt doesn't happen by accident. Something specific was going on, and until I knew what, the result was a number rather than a finding.&lt;/p&gt;

&lt;p&gt;This post is the investigation. It's the part of the project I'm proudest of, because the interesting answer only appeared after the obvious ones had been ruled out properly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 0: is it even real?
&lt;/h2&gt;

&lt;p&gt;My first instinct was that the result was a bug in &lt;em&gt;my&lt;/em&gt; code, not a property of the framework.&lt;br&gt;
0.323 vs 0.614 on identical inputs smells like a parsing or logging problem.&lt;/p&gt;

&lt;p&gt;So before any theory, I checked the boring explanations:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Was the trace logging broken?&lt;/strong&gt; No. Every SDK answer had a non-empty retrieval trace. The agent did search, and it did get results.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Were the prompts different?&lt;/strong&gt; No. The two agent prompts are byte-identical.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Was the scorer misreading answers?&lt;/strong&gt; The scorer looks for citations in the required &lt;code&gt;[url]&lt;/code&gt; format. 66 of the 127 SDK answers had no citation in that format, in otherwise well-formed prose.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So the effect was real at the level of the outputs. Now: why?&lt;/p&gt;
&lt;h2&gt;
  
  
  Writing the plan down first
&lt;/h2&gt;

&lt;p&gt;There were several plausible causes, and I had a favourite. That's exactly when it's easiest to fool yourself: you keep running variations until one confirms the idea you already had.&lt;/p&gt;

&lt;p&gt;So before running anything, I wrote the plan into the project's decision log:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One arm:&lt;/strong&gt; OpenAI Agents SDK × gpt-4o-mini, all 127 questions, same eval set, same prompt, same scorer, same citation format.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One change per run&lt;/strong&gt;, each compared against the run before it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One rule for "this helped":&lt;/strong&gt; the paired bootstrap 95% confidence interval of the per-question CSR difference (10,000 resamples, fixed seed) must exclude zero.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Every result gets reported, whichever way it comes out.&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Paired is the important word there. Every run answers the &lt;em&gt;same&lt;/em&gt; 127 questions, so the right comparison is question by question, which removes most of the noise from questions simply being easier or harder.&lt;/p&gt;
&lt;h2&gt;
  
  
  Hypothesis 1: the agent loop
&lt;/h2&gt;

&lt;p&gt;This was my favourite. Agents search, read, maybe search again, then write. By the time the final answer is written, the sources from the first tool call are many messages back. Surely they get lost along the way?&lt;/p&gt;

&lt;p&gt;It didn't survive contact with the data I already had:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;LangGraph runs the same kind of loop and lost nothing.&lt;/strong&gt; Its difference from single-shot RAG, question by question, was +0.000.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The agents barely looped.&lt;/strong&gt; The SDK agent made about 1.3 searches per answer, LangGraph about 1.2. There wasn't much of a "long multi-step context" for sources to get lost in.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So the loop wasn't it. That left the question of what was different between two agents doing nearly the same thing.&lt;/p&gt;
&lt;h2&gt;
  
  
  Hypothesis 2: temperature (a confound I found in my own code)
&lt;/h2&gt;

&lt;p&gt;While writing the plan I read both pipelines line by line, and found something I'd missed: &lt;strong&gt;the SDK pipeline never set a temperature.&lt;/strong&gt; The SDK's model settings leave it unset, so the API default was used. The LangGraph pipeline ran at temperature 0.&lt;/p&gt;

&lt;p&gt;So "same model, same prompt" was true, but "same settings" wasn't. A more random sampler could plausibly be sloppier about formatting. This had to be tested before anything else, because if it&lt;br&gt;
explained the gap, the headline was an artifact of my setup.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Run&lt;/th&gt;
&lt;th&gt;Change&lt;/th&gt;
&lt;th&gt;CSR&lt;/th&gt;
&lt;th&gt;No &lt;code&gt;[url]&lt;/code&gt; citation&lt;/th&gt;
&lt;th&gt;vs previous&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;E0&lt;/td&gt;
&lt;td&gt;original config, re-run&lt;/td&gt;
&lt;td&gt;0.333&lt;/td&gt;
&lt;td&gt;60/127&lt;/td&gt;
&lt;td&gt;vs published 0.323: −0.011, &lt;strong&gt;replicates&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;E1&lt;/td&gt;
&lt;td&gt;+ temperature 0&lt;/td&gt;
&lt;td&gt;0.412&lt;/td&gt;
&lt;td&gt;54/127&lt;/td&gt;
&lt;td&gt;+0.082 [−0.010, +0.175], &lt;strong&gt;not significant&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two useful results. First, the headline replicated: a fresh run of the original setup scored 0.333 against the published 0.323. Second, matching the temperature helped a little, but not significantly, and E1 was still clearly below single-shot RAG (−0.207, a real difference).&lt;/p&gt;

&lt;p&gt;Temperature was a genuine flaw in my setup, and fixing it was right. But it wasn't the cause.&lt;/p&gt;
&lt;h2&gt;
  
  
  Hypothesis 3: the tool description
&lt;/h2&gt;

&lt;p&gt;The MCP tool comes with a description the model reads. A real hosted MCP retrieval product publishes recommended wording for it, so I tested that next (E2): same as E1, with the recommended tool description served by the MCP server.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Run&lt;/th&gt;
&lt;th&gt;Change&lt;/th&gt;
&lt;th&gt;CSR&lt;/th&gt;
&lt;th&gt;No &lt;code&gt;[url]&lt;/code&gt; citation&lt;/th&gt;
&lt;th&gt;vs E1&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;E2&lt;/td&gt;
&lt;td&gt;E1 + recommended tool description&lt;/td&gt;
&lt;td&gt;0.382&lt;/td&gt;
&lt;td&gt;54/127&lt;/td&gt;
&lt;td&gt;−0.040, &lt;strong&gt;no effect&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Nothing. That fits the product's own documentation once I read it at the source: it puts the citation &lt;em&gt;rules&lt;/em&gt; in the system prompt, and the recommended tool description only says that chunks carry a source URL. (An earlier summary of a blog post had told me the opposite. Lesson: read the primary source.)&lt;/p&gt;
&lt;h2&gt;
  
  
  Hypothesis 4: the API underneath
&lt;/h2&gt;

&lt;p&gt;This is where it got interesting. The OpenAI Agents SDK talks to OpenAI's &lt;strong&gt;Responses API&lt;/strong&gt; by default. LangGraph's OpenAI integration uses the older &lt;strong&gt;Chat Completions API&lt;/strong&gt;. Same model name, different API, and potentially a different way of delivering instructions and tool outputs to the model.&lt;/p&gt;

&lt;p&gt;Before running the experiment I wanted to rule out the transport itself. Maybe the SDK was reformatting tool results and dropping the URLs before the model ever saw them? I added a flag to dump exactly what the model receives. (It only got through one question before my OpenAI credits ran out, which is its own lesson.) On that question, the system prompt arrived intact, and the tool output arrived as one block of JSON, about 12,000 characters, with every chunk's &lt;code&gt;source_url&lt;/code&gt; intact. The SDK wasn't hiding the sources.&lt;/p&gt;

&lt;p&gt;Then the experiment (E3): same as E1, but the SDK agent uses Chat Completions.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Run&lt;/th&gt;
&lt;th&gt;Change&lt;/th&gt;
&lt;th&gt;CSR&lt;/th&gt;
&lt;th&gt;No &lt;code&gt;[url]&lt;/code&gt; citation&lt;/th&gt;
&lt;th&gt;vs E1&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;E3&lt;/td&gt;
&lt;td&gt;E1 + Chat Completions API&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.533&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;18/127&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+0.108 [+0.010, +0.206], real&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;single-shot RAG, for reference&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;0.618&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;em&gt;19/127&lt;/em&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;That looked like the one. Answers without a &lt;code&gt;[url]&lt;/code&gt; citation dropped from 54 to 18, the same level as single-shot RAG. (Hold on to the words "without a &lt;code&gt;[url]&lt;/code&gt; citation". They turn out to matter.)&lt;/p&gt;
&lt;h2&gt;
  
  
  Hypothesis 5: following the published guidance in full
&lt;/h2&gt;

&lt;p&gt;One more arm, added (and written down) before running it: what if a developer follows the retrieval provider's published guidance completely, meaning their recommended system prompt plus tool description?&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Run&lt;/th&gt;
&lt;th&gt;Change&lt;/th&gt;
&lt;th&gt;CSR&lt;/th&gt;
&lt;th&gt;vs E1&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;E4&lt;/td&gt;
&lt;td&gt;E1 + recommended system prompt + tool description&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.515&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+0.111 [+0.015, +0.212], real&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A real improvement by this measure, and enough that the SDK was no longer significantly below single-shot RAG. One caveat: that prompt asks for a different citation format (&lt;code&gt;[[title](url)]&lt;/code&gt;), so this run changes the wording and the format together and can't separate them.&lt;/p&gt;
&lt;h2&gt;
  
  
  The twist: reading the "uncited" answers
&lt;/h2&gt;

&lt;p&gt;At this point I had a clean story: the default API loses citations, and switching APIs (or using a good system prompt) recovers them.&lt;/p&gt;

&lt;p&gt;Then I did what I should have done much earlier, and read the 66 "uncited" answers from the published run one by one.&lt;/p&gt;

&lt;p&gt;Most of them weren't uncited. They looked like this:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;...the response model used for returning user data (like &lt;code&gt;UserOut&lt;/code&gt;) specifically does not&lt;br&gt;
include the password or any hashed password fields. &lt;a href="https://fastapi.tiangolo.com/tutorial/security/simple-oauth2/" rel="noopener noreferrer"&gt;source_url&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The right page, right after the claim, just written as a Markdown link instead of the required &lt;code&gt;[url]&lt;/code&gt;. Counting them up:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Published SDK run, 66 answers with no &lt;code&gt;[url]&lt;/code&gt; citation&lt;/th&gt;
&lt;th&gt;Count&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;URL present as a Markdown link &lt;code&gt;[text](url)&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;51&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;URL present in another form, e.g. &lt;code&gt;[source: https://...]&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No URL anywhere&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For comparison, single-shot RAG has 19 answers with no URL anywhere. On the crude question "does the answer mention any URL at all", the SDK agent was no worse than the baseline. (Whether it cited the &lt;em&gt;right&lt;/em&gt; sources as often is a separate question, and it took three runs to answer it; see below.)&lt;/p&gt;

&lt;p&gt;And the same breakdown across the experiments shows what the API switch actually fixed:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Run&lt;/th&gt;
&lt;th&gt;No &lt;code&gt;[url]&lt;/code&gt; citation&lt;/th&gt;
&lt;th&gt;of which Markdown link&lt;/th&gt;
&lt;th&gt;other URL form&lt;/th&gt;
&lt;th&gt;no URL at all&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;E0 default (Responses API)&lt;/td&gt;
&lt;td&gt;60&lt;/td&gt;
&lt;td&gt;38&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;15&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;E1 temperature 0&lt;/td&gt;
&lt;td&gt;54&lt;/td&gt;
&lt;td&gt;41&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;11&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;E3 Chat Completions&lt;/td&gt;
&lt;td&gt;18&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LangGraph (Chat Completions)&lt;/td&gt;
&lt;td&gt;15&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The answers with no URL at all barely changed (11 → 9). What collapsed was the Markdown links (41 → 9). &lt;strong&gt;Chat Completions didn't make the model cite more. It made the model follow the citation&lt;br&gt;
format it was asked for.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;So the accurate version of the finding is this:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;With the same model and prompt, the OpenAI Agents SDK's default (the Responses API) ignored the&lt;br&gt;
required citation format in 51 of 127 answers (40%), mostly writing Markdown links instead.&lt;br&gt;
On Chat Completions that falls to 9, close to LangGraph (8) and single-shot RAG (0).&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Is that less important than "the agent forgot its sources"? Not for anyone building a product.&lt;br&gt;
Every system that consumes citations does it by format: the "view sources" panel, the click-through tracking, the guardrail, the eval harness. When the format breaks, the citation is gone &lt;em&gt;as far as the system is concerned&lt;/em&gt;. And unlike "the agent forgot", it's specific, testable and fixable.&lt;/p&gt;

&lt;p&gt;But it also meant every CSR number above was measuring two things at once: whether the agent cited the right source, and whether it used the exact format my parser recognised. I had to separate&lt;br&gt;
them.&lt;/p&gt;

&lt;p&gt;You can reproduce the breakdown yourself from the committed outputs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;

&lt;span class="n"&gt;CITE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;\[(https?://[^\]\s]+)\]&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# the benchmark's required format
&lt;/span&gt;
&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;results/openai_agent_openai.jsonl&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
&lt;span class="n"&gt;uncited&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;answer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;CITE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;answer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])]&lt;/span&gt;
&lt;span class="n"&gt;md_link&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;\]\(https?://&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;uncited&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;no_url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;https?://&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;uncited&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;uncited&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;md_link&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;no_url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# 66 51 8
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Re-scoring with every format counted
&lt;/h2&gt;

&lt;p&gt;I wrote the new rule into the decision log before re-running anything: count &lt;code&gt;[url]&lt;/code&gt;, Markdown links &lt;code&gt;[text](url)&lt;/code&gt; and the &lt;code&gt;[[title](url)]&lt;/code&gt; bracket format as citations; don't count bare URLs in running text. Then I re-scored all ten runs, with both the original entailment model and an independent gpt-4o judge (chosen because it agreed best with my own blind judgements).&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Comparison (any format counted, first run)&lt;/th&gt;
&lt;th&gt;Original scorer&lt;/th&gt;
&lt;th&gt;gpt-4o judge&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;SDK (default) − single-shot RAG&lt;/td&gt;
&lt;td&gt;−0.163, &lt;strong&gt;significant&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;−0.102, not significant&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SDK at temperature 0 − single-shot&lt;/td&gt;
&lt;td&gt;−0.056, not significant&lt;/td&gt;
&lt;td&gt;−0.045, not significant&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Chat Completions (E3) − temperature 0 (E1)&lt;/td&gt;
&lt;td&gt;+0.020, not significant&lt;/td&gt;
&lt;td&gt;+0.069, not significant&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Published prompt setup (E4) − E1&lt;/td&gt;
&lt;td&gt;+0.030, not significant&lt;/td&gt;
&lt;td&gt;+0.035, not significant&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Three conclusions from that run, and they're less exciting than the ones above:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;One run couldn't settle whether citations were really lost.&lt;/strong&gt; The remaining gap was significant under one scorer and not the other, and it disappeared at temperature 0. At the time I wrote that a real loss wasn't established. (The next section is what settled it.)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The API switch and the published prompt fixed the format, not the citations.&lt;/strong&gt; Counted in any format, neither changed whether sources got cited (measured on this run only). What they change is how often the model follows the requested format: 41 wrong-format answers drop to 9 (Chat Completions) or 20 (the published setup).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The robust result is the format drift itself&lt;/strong&gt;, counted directly from the answers with no scorer involved: 51 of 127 on the SDK default, 0 on single-shot RAG.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;I'd like to say I caught this before telling anyone. I didn't: I'd already sent the "loses half" version to people whose opinion I cared about, and had to follow up the same day with a correction. That was uncomfortable, and it was still the right call. (As the next section shows, the correction itself leaned too far the other way.)&lt;/p&gt;

&lt;h2&gt;
  
  
  One run couldn't settle it, so I ran everything three times
&lt;/h2&gt;

&lt;p&gt;Everything above is one run per setup. The confidence intervals cover which questions happened to be in the set, but not how much a fresh set of generations moves the numbers. With a gap that was significant under one scorer and not the other, that run-to-run variation was exactly what I didn't know.&lt;/p&gt;

&lt;p&gt;So, same habit as before, I wrote the rule down first: a difference counts as robust only if it has the same sign in all three runs &lt;strong&gt;and&lt;/strong&gt; the pooled paired 95% interval excludes zero. Then I&lt;br&gt;
ran every gpt-4o-mini setup twice more with fresh generations.&lt;/p&gt;

&lt;p&gt;The format drift first, counted straight from the answers:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Wrong-format answers (cite only in another link format)&lt;/th&gt;
&lt;th&gt;Run 1&lt;/th&gt;
&lt;th&gt;Run 2&lt;/th&gt;
&lt;th&gt;Run 3&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Single-shot RAG&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LangGraph agent&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;12&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenAI Agents SDK, defaults&lt;/td&gt;
&lt;td&gt;51&lt;/td&gt;
&lt;td&gt;53&lt;/td&gt;
&lt;td&gt;46&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SDK + temperature 0 (E1)&lt;/td&gt;
&lt;td&gt;41&lt;/td&gt;
&lt;td&gt;60&lt;/td&gt;
&lt;td&gt;60&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SDK + Chat Completions (E3)&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;12&lt;/td&gt;
&lt;td&gt;15&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SDK + published prompt setup (E4)&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;23&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The drift is stable, and Chat Completions and the published setup cut it in every run. Temperature 0 on its own doesn't.&lt;/p&gt;

&lt;p&gt;Then the question one run couldn't answer, with every format counted:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Comparison (any format, pooled over 3 runs)&lt;/th&gt;
&lt;th&gt;Original scorer&lt;/th&gt;
&lt;th&gt;gpt-4o judge&lt;/th&gt;
&lt;th&gt;Verdict&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;SDK (default) − single-shot RAG&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;−0.120&lt;/strong&gt; [−0.209, −0.030]&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;−0.086&lt;/strong&gt; [−0.164, −0.005]&lt;/td&gt;
&lt;td&gt;Robust under both&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SDK at temperature 0 − single-shot&lt;/td&gt;
&lt;td&gt;−0.066 [−0.160, +0.028]&lt;/td&gt;
&lt;td&gt;−0.045 [−0.130, +0.043]&lt;/td&gt;
&lt;td&gt;No detectable loss&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LangGraph − single-shot&lt;/td&gt;
&lt;td&gt;+0.023&lt;/td&gt;
&lt;td&gt;+0.037&lt;/td&gt;
&lt;td&gt;No difference&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Per run, the SDK's gap was −0.163, −0.143 and −0.055 under the original scorer (−0.102, −0.087 and −0.045 under the judge). Same sign every time, under both scorers. So the SDK's default setup&lt;br&gt;
&lt;em&gt;does&lt;/em&gt; lose some citations for real. It's just much smaller than it first looked: counting only the requested format, the pooled gap is −0.302 [−0.394, −0.210]. &lt;strong&gt;About 60% of the apparent loss is format, and about 40% is a real, smaller loss of citations.&lt;/strong&gt; At temperature 0, no loss is detectable.&lt;/p&gt;

&lt;p&gt;Why one run couldn't settle it: the per-run gap ranged from −0.05 to −0.16. A single run can land anywhere in that range, and my first run happened to be the largest. The version I first sent overclaimed from it; my same-day correction, which called the gap not significant, underclaimed. The honest answer was in between, and it took three runs to see it.&lt;/p&gt;

&lt;h2&gt;
  
  
  A fix the retrieval server controls
&lt;/h2&gt;

&lt;p&gt;Chat Completions and a good system prompt both reduce the drift, but both live in the customer's agent. A retrieval provider can't set them. So I tested a fix on the server side (pre-registered,&lt;br&gt;
three runs per agent, on my self-hosted mirror server): the search tool tags each chunk with a short, stable source ID like &lt;code&gt;[S3fa21c]&lt;/code&gt;, the tool description and prompt ask for &lt;code&gt;[ID]&lt;/code&gt; citations, and a small deterministic step maps each ID back to its URL after the answer is&lt;br&gt;
written, accepting only IDs from that answer's own retrieved chunks.&lt;/p&gt;

&lt;p&gt;On the SDK agent, wrong-format answers fell from 51 / 53 / 46 to &lt;strong&gt;5 / 5 / 4&lt;/strong&gt;, without the agent citing less. Any-format CSR rose by +0.096 (original scorer) and +0.125 (gpt-4o judge), and the&lt;br&gt;
gap to single-shot RAG closed (−0.028 / +0.042, not significant). The details and the caveats (tool output and prompt changed together, no human audit of those citations yet) are in Part 6 (coming soon).&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd do differently
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Read the raw outputs before building the theory, and certainly before sharing a number.&lt;/strong&gt; I had the 66 answers from day one. A twenty-minute read would have pointed at format compliance before I&lt;br&gt;
spent days on mechanism experiments, and before the overstated version left my laptop. The experiments were still worth it (they found the API effect on format and ruled out the loop and the tool description), but I'd have framed them correctly from the start.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Treat the parser as part of the metric.&lt;/strong&gt; My scorer was validated against human judgement, three ways. All three checks shared the same citation parser, so none of them could catch this.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Check "same settings" line by line.&lt;/strong&gt; The temperature confound sat in my code for two months.&lt;br&gt;
It didn't change the conclusion, but I only know that because I tested it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Run a close comparison more than once before calling it either way.&lt;/strong&gt; One run made me overclaim first and underclaim second.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways for agent builders
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Your framework's default model API is part of your system's behaviour.&lt;/strong&gt; Two agents with the same model name followed the same citation instruction very differently.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Test format compliance explicitly.&lt;/strong&gt; If anything downstream parses model output by format (citations, JSON, tags), count how often the format is followed, per framework and per API.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Put the rules where they work.&lt;/strong&gt; A system prompt with explicit citation rules halved the format drift (41 → 20). The same intent in a tool description did nothing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Measure format and presence separately.&lt;/strong&gt; "No citation in the format I parse" and "no citation at all" are different failures with different fixes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Write the experiment plan down before you run it.&lt;/strong&gt; It's cheap, and it's the difference between "I found the cause" and "I kept trying things until one worked".&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;Limits:&lt;/strong&gt; three runs per gpt-4o-mini setup (the tool-description run, E2, only once), one model (gpt-4o-mini), one corpus, and a self-hosted mirror of an MCP server rather than anyone's production system. The API switch and the published prompt were checked against any-format CSR on one run only. Whether the Responses API format drift holds for other models, or other agent frameworks, is the obvious next experiment.&lt;/p&gt;




&lt;h2&gt;
  
  
  The series: Citations at the MCP Boundary
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;a href="https://dev.to/ayanarshad02/your-mcp-server-returns-the-source-does-the-answer-still-cite-it-26n8"&gt;Your MCP Server Returns the Source. Does the Answer Still Cite It?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One Agent Framework Seemed to Lose Half Its Citations. Most Were Just Reformatted.&lt;/strong&gt; (you are here)&lt;/li&gt;
&lt;li&gt;When Your Eval Grades Its Own Homework &lt;em&gt;(coming soon)&lt;/em&gt;
&lt;/li&gt;
&lt;li&gt;I Audited My Own Benchmark Like a Hostile Reviewer. It Had Five Bugs. &lt;em&gt;(coming soon)&lt;/em&gt;
&lt;/li&gt;
&lt;li&gt;Before You Measure an Agent, Fix Your Corpus: 20% of My Chunks Had No Code &lt;em&gt;(coming soon)&lt;/em&gt;
&lt;/li&gt;
&lt;li&gt;Serving RAG Over MCP? A Practical Playbook for Keeping Your Citations &lt;em&gt;(coming soon)&lt;/em&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;Previous:&lt;/strong&gt; &lt;a href="https://dev.to/ayanarshad02/your-mcp-server-returns-the-source-does-the-answer-still-cite-it-26n8"&gt;Part 1: Your MCP Server Returns the Source. Does the Answer Still Cite It?&lt;/a&gt;&lt;br&gt;
&lt;strong&gt;Next:&lt;/strong&gt; Part 3: When Your Eval Grades Its Own Homework &lt;em&gt;(coming soon)&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A new part is published every week.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Repo:&lt;/strong&gt; &lt;a href="https://github.com/AyanArshad02/csr-mcp-benchmark" rel="noopener noreferrer"&gt;https://github.com/AyanArshad02/csr-mcp-benchmark&lt;/a&gt; (&lt;code&gt;results/sdk_ablation_report.txt&lt;/code&gt;,&lt;br&gt;
&lt;code&gt;results/anyformat_report.txt&lt;/code&gt;, &lt;code&gt;results/format_compliance.txt&lt;/code&gt; and &lt;code&gt;results/repeats_report.txt&lt;/code&gt;&lt;br&gt;
have every number in this post).&lt;/p&gt;

&lt;p&gt;&lt;em&gt;I'm an AI engineer who builds production RAG and agent systems and the evals that prove they work, and I'm looking for a remote AI engineering role at a US or EU startup. If that's your team, I'd love to talk.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>openai</category>
      <category>mcp</category>
    </item>
    <item>
      <title>Your MCP Server Returns the Source. Does the Answer Still Cite It?</title>
      <dc:creator>Md Ayan Arshad</dc:creator>
      <pubDate>Sat, 17 Oct 2026 07:26:58 +0000</pubDate>
      <link>https://dev.to/ayanarshad02/your-mcp-server-returns-the-source-does-the-answer-still-cite-it-26n8</link>
      <guid>https://dev.to/ayanarshad02/your-mcp-server-returns-the-source-does-the-answer-still-cite-it-26n8</guid>
      <description>&lt;p&gt;&lt;em&gt;A benchmark for what happens to citations after a retrieval server hands its results to an AI&lt;br&gt;
agent it doesn't control.&lt;/em&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Part 1 of 6 in the series &lt;em&gt;Citations at the MCP Boundary&lt;/em&gt;.&lt;/strong&gt; This is where the series starts. The parts after this one go deeper into each finding; the full list is at the end of this post.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Here's a situation that's becoming normal very quickly.&lt;/p&gt;

&lt;p&gt;A company turns its documentation into a search tool. It exposes that tool over MCP (the Model Context Protocol), so any agent can use it: Claude, ChatGPT, a coding assistant in someone's IDE, an agent a customer built with LangGraph or the OpenAI Agents SDK. The tool does its job well. For every query it returns the most relevant chunks of documentation, and each chunk comes with its source URL:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"source_url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://fastapi.tiangolo.com/tutorial/body-updates/"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"# Body - Updates ..."&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then the agent takes over. It reads the chunks, decides what matters and writes the final answer.&lt;/p&gt;

&lt;p&gt;The retrieval server never sees that answer. So the provider can be sure the right source went &lt;em&gt;in&lt;/em&gt;, but has no idea whether a citation comes &lt;em&gt;out&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;For a documentation assistant that's a real problem. An answer about your API that's correct but uncited can't be checked, and the user can't click through to the page that explains the edge cases. An answer that cites the wrong page is worse: it looks verified when it isn't.&lt;/p&gt;

&lt;p&gt;I couldn't find anyone measuring this, so I built a benchmark for it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The question
&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;When a RAG knowledge base is consumed by an AI agent over MCP, instead of a single retrieve-then-answer call, do the citations survive?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The obvious hypothesis was that agents lose citations. An agent loops: it searches, reads, maybe searches again, and then writes. The sources from the first call end up far back in the context by the time the answer is written. It seemed natural that they'd get lost along the way.&lt;/p&gt;

&lt;p&gt;That hypothesis turned out to be mostly wrong, and the way it was wrong is more useful than the hypothesis itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;I built a complete, reproducible copy of this boundary:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The corpus:&lt;/strong&gt; the FastAPI documentation, 153 pages split into 2,085 heading-aware chunks. FastAPI has clean Markdown, stable public URLs and is widely used, so anyone can check the work.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The MCP server:&lt;/strong&gt; one tool, &lt;code&gt;search_fastapi_knowledge_sources&lt;/code&gt;, returning exactly
&lt;code&gt;{source_url, content}&lt;/code&gt; over streamable HTTP. It mirrors the contract of a real hosted MCP
retrieval product, down to the naming pattern.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The questions:&lt;/strong&gt; 250 drafted, &lt;strong&gt;127 kept after I checked every one by hand&lt;/strong&gt; against the actual
docs page: 82 answerable from one page, 29 that need two or more pages, and 16 the docs don't
cover at all (to catch made-up answers).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Three pipelines, held as equal as possible:&lt;/strong&gt; the same model (gpt-4o-mini), the same docs, the
same system prompt and the same required citation format (&lt;code&gt;[url]&lt;/code&gt; right after each claim):

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Single-shot RAG.&lt;/strong&gt; Search once, then answer. This is the control.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A LangGraph agent.&lt;/strong&gt; A ReAct agent that decides when and how often to call the MCP tool.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;An OpenAI Agents SDK agent.&lt;/strong&gt; The same idea, built with OpenAI's agent framework and its
built-in MCP support.&lt;/li&gt;
&lt;/ol&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The metric: Citation Survival Rate
&lt;/h2&gt;

&lt;p&gt;Measuring "does it cite" turned out to need more care than I expected. An answer can be full of URLs and still be badly attributed. It can cite a real page that doesn't support the sentence, or cite a plausible FastAPI URL the agent never actually retrieved (models know what FastAPI URLs look like).&lt;/p&gt;

&lt;p&gt;So &lt;strong&gt;CSR (Citation Survival Rate)&lt;/strong&gt; requires every citation to pass two checks:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The support check.&lt;/strong&gt; Does the cited chunk actually support the claim? Each answer is split
into small atomic claims (the FActScore approach), and an entailment model checks each claim
against the chunk it cites (the ALCE approach).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The trace check.&lt;/strong&gt; Was that URL really in what the agent retrieved? Every tool call is logged, so a citation to a page the agent never saw is caught as fabricated.&lt;/li&gt;
&lt;/ol&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;CSR&lt;/strong&gt; = of the correct sources the agent actually retrieved, the fraction that end up in the final answer, cited on a claim the source genuinely supports.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Notice the condition: &lt;em&gt;the sources the agent actually retrieved&lt;/em&gt;. If the agent never found the right page, that's a retrieval failure, and CSR leaves it out. CSR only measures what happens after the right source has crossed the boundary.&lt;/p&gt;

&lt;h2&gt;
  
  
  The results, and the mistake I almost published
&lt;/h2&gt;

&lt;p&gt;Here's what the metric said on the first pass:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Pipeline (gpt-4o-mini)&lt;/th&gt;
&lt;th&gt;CSR&lt;/th&gt;
&lt;th&gt;95% CI&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Single-shot RAG&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.618&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;[0.528, 0.708]&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LangGraph agent&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.614&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;[0.524, 0.705]&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenAI Agents SDK agent&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.323&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;[0.237, 0.414]&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The LangGraph agent matched single-shot RAG (a difference of +0.000 question by question), so an agent loop by itself didn't cost citations. The OpenAI Agents SDK agent seemed to lose about half: in &lt;strong&gt;66 of its 127 answers there was no citation in the &lt;code&gt;[url]&lt;/code&gt; format the prompt asked for.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;I took that at face value for longer than I should have. When I finally read those 66 answers one by one, &lt;strong&gt;most of them did cite their sources&lt;/strong&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;51 cited the URL right after the claim, but as a Markdown link: &lt;code&gt;[source](https://...)&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;7 used another form, like &lt;code&gt;[source: https://...]&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;only 8 had no URL at all (single-shot RAG has 19 of those)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;My scorer only recognised the requested &lt;code&gt;[url]&lt;/code&gt; format, so it counted every one of those Markdown links as a missing citation. I fixed the definition (written down before re-scoring: &lt;code&gt;[url]&lt;/code&gt;, Markdown links and the other bracket format all count) and re-scored everything:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Pipeline (gpt-4o-mini)&lt;/th&gt;
&lt;th&gt;Only the requested format counted&lt;/th&gt;
&lt;th&gt;Any citation format counted&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Single-shot RAG&lt;/td&gt;
&lt;td&gt;0.618&lt;/td&gt;
&lt;td&gt;0.618&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LangGraph agent&lt;/td&gt;
&lt;td&gt;0.614&lt;/td&gt;
&lt;td&gt;0.643&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenAI Agents SDK agent&lt;/td&gt;
&lt;td&gt;0.323&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.470&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Counted in any format, the SDK's remaining gap to single-shot on that run was −0.16 under the original scorer (significant) but −0.10 under a gpt-4o judge (not significant). One run couldn't settle it, so I ran every setup three times. A smaller loss held in every run and under both&lt;br&gt;
scorers: &lt;strong&gt;−0.12&lt;/strong&gt; (original scorer) and &lt;strong&gt;−0.09&lt;/strong&gt; (gpt-4o judge), pooled, against about −0.30 counting only the requested format. So the drop is &lt;strong&gt;about 60% format and 40% a real, smaller loss of citations.&lt;/strong&gt; At temperature 0 no loss is detectable.&lt;/p&gt;

&lt;p&gt;The bigger and more stable effect is simpler:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;The OpenAI Agents SDK, on its default API, ignored the requested citation format in 51 of 127&lt;br&gt;
answers (40%), and 53 and 46 in the two repeat runs. Single-shot RAG did that 0 times; the&lt;br&gt;
LangGraph agent 8 times.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That matters more than it sounds. Anything downstream that reads citations in a fixed format (a docs widget, a "view sources" panel, a guardrail, an eval like mine) sees nothing at all. The citation is gone &lt;em&gt;as far as the system is concerned&lt;/em&gt;, even though a human reading the text could find it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why? Not the reason I expected
&lt;/h2&gt;

&lt;p&gt;Part 2 has the full investigation. The short version: I ran five controlled experiments on the SDK agent, changing one thing at a time, with the plan written down &lt;em&gt;before&lt;/em&gt; running anything:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Not the agent loop.&lt;/strong&gt; LangGraph loops too and kept the format. The agents also barely looped: about 1.2–1.3 searches per answer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not temperature, at least not alone.&lt;/strong&gt; The SDK pipeline had never set a temperature (LangGraph used 0). Matching it barely changed the format drift (60 → 54 answers without &lt;code&gt;[url]&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not the transport.&lt;/strong&gt; I dumped exactly what the model receives through the SDK: every chunk arrives with its &lt;code&gt;source_url&lt;/code&gt; intact, and 98 of the 118 Markdown links point to pages the agent retrieved. The sources make it across the boundary.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The API.&lt;/strong&gt; The OpenAI Agents SDK calls OpenAI's &lt;strong&gt;Responses API&lt;/strong&gt; by default; LangGraph's OpenAI integration uses &lt;strong&gt;Chat Completions&lt;/strong&gt;. Switching the same SDK agent to Chat Completions cut the wrong-format answers from &lt;strong&gt;41 to 9&lt;/strong&gt;. A published system prompt for grounded answers cut them to 20. Both held across three runs. Neither made a significant difference to whether sources were cited at all (measured on one run): they fix the format.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A fix on the server side.&lt;/strong&gt; Both of those live in the agent, which a retrieval provider doesn't control. So I tested one the server can make: tag every chunk with a short source ID (&lt;code&gt;[S3fa21c]&lt;/code&gt;), ask for &lt;code&gt;[ID]&lt;/code&gt; citations, and map IDs back to URLs after the answer is written. On my self-hosted mirror, wrong-format answers fell from about 50 to about 5 per run, the SDK agent's CSR rose by 0.10–0.13, and its gap to single-shot closed. (Part 6 has the numbers and the caveats, including that no human has audited those citations yet.)&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Can you trust these numbers?
&lt;/h2&gt;

&lt;p&gt;A benchmark is only as good as its scorer, so I spent as long checking the measurement as running the experiment:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Two kinds of scorer.&lt;/strong&gt; An entailment model and an LLM judge (gpt-4o), chosen by benchmarking checkers against 115 of my own blind judgements; the judge agreed with me best (κ 0.86). Where they disagree, I say so.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Checked against a human.&lt;/strong&gt; On 30 blind cases, CSR's verdicts agreed with mine on 25 to 27, depending on the scorer. (Two of those cases had the same Markdown-link blind spot, which is part of how I found it.)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Repeated.&lt;/strong&gt; Every gpt-4o-mini setup ran three times, with a robustness rule written down first: same sign in every run, and a pooled interval that excludes zero.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Audited before publishing.&lt;/strong&gt; Going through the pipeline as a hostile reviewer would, I found five bugs, four of which would otherwise have shipped. (That story is Part 4.) The format issue above is the sixth, and the one I'm least proud of: I'd already sent the overstated number to people before I caught it, and had to correct it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And the limits, stated plainly: one corpus, two models, three runs per gpt-4o-mini setup (one for the Claude runs), small human checks, and a self-hosted mirror of an MCP server rather than anyone's production system.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd take away from this
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;If you serve RAG through MCP, your accuracy guarantee stops at the boundary.&lt;/strong&gt; You control the retrieval. You don't control the agent that writes the answer. If citations matter to your product, measure what happens on the other side.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Framework defaults are part of your system.&lt;/strong&gt; Two agents with the same model and the same prompt followed the citation format very differently because of one default setting a developer might never look at.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Measure format and presence separately.&lt;/strong&gt; "No citation in the format I parse" and "no citation" are different failures with different fixes. Count both.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Read the outputs, not just the metric.&lt;/strong&gt; A count of "uncited answers" hid the fact that most of them did contain the source. Reading 66 answers changed the conclusion, and should have happened before I told anyone the number.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Run it more than once.&lt;/strong&gt; One run of the remaining gap could land anywhere from −0.05 to −0.16. Three runs showed it was real, and smaller than it first looked.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The retrieval side can fix the format.&lt;/strong&gt; Give the agent a short source ID to copy instead of a long URL to reformat, and map it back yourself.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  The series: Citations at the MCP Boundary
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Your MCP Server Returns the Source. Does the Answer Still Cite It?&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="//./02-one-framework-seemed-to-lose-half-its-citations.md"&gt;One Agent Framework Seemed to Lose Half Its Citations. Most Were Just Reformatted.&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="//./03-when-your-eval-grades-its-own-homework.md"&gt;When Your Eval Grades Its Own Homework&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="//./04-five-bugs-before-publishing.md"&gt;I Audited My Own Benchmark Like a Hostile Reviewer. It Had Five Bugs.&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="//./05-building-the-measuring-instrument.md"&gt;Before You Measure an Agent, Fix Your Corpus: 20% of My Chunks Had No Code&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="//./06-serving-rag-over-mcp-playbook.md"&gt;Serving RAG Over MCP? A Practical Playbook for Keeping Your Citations&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;Next:&lt;/strong&gt; &lt;a href="//./02-one-framework-seemed-to-lose-half-its-citations.md"&gt;Part 2: One Agent Framework Seemed to Lose Half Its Citations. Most Were Just Reformatted.&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A new part is published every week.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Repo, traces and every number:&lt;/strong&gt; &lt;a href="//github.com/AyanArshad02/csr-mcp-benchmark"&gt;csr-mcp-benchmark&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;I'm an AI engineer who builds production RAG and agent systems and the evals that prove they work. I'm looking for a remote AI engineering role at a US or EU startup. If your team is building with RAG, agents or MCP, I'd love to talk.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>rag</category>
      <category>mcp</category>
      <category>llm</category>
    </item>
    <item>
      <title>wrote article about why increasing retrieval from top 5 to top 20 worsens my answer quality: https://dev.to/ayanarshad02/i-increased-retrieval-from-top-5-to-top-20-my-answers-got-worse-3mke</title>
      <dc:creator>Md Ayan Arshad</dc:creator>
      <pubDate>Fri, 08 May 2026 03:21:48 +0000</pubDate>
      <link>https://dev.to/ayanarshad02/wrote-article-about-why-increasing-retrieval-from-top-5-to-top-20-worsens-my-answer-quality-4ffn</link>
      <guid>https://dev.to/ayanarshad02/wrote-article-about-why-increasing-retrieval-from-top-5-to-top-20-worsens-my-answer-quality-4ffn</guid>
      <description>&lt;div class="ltag__link--embedded"&gt;
  &lt;div class="crayons-story "&gt;
  &lt;a href="https://dev.to/ayanarshad02/i-increased-retrieval-from-top-5-to-top-20-my-answers-got-worse-3mke" class="crayons-story__hidden-navigation-link"&gt;I Increased Retrieval From Top-5 to Top-20. My Answers Got Worse&lt;/a&gt;


  &lt;div class="crayons-story__body crayons-story__body-full_post"&gt;
    &lt;div class="crayons-story__top"&gt;
      &lt;div class="crayons-story__meta"&gt;
        &lt;div class="crayons-story__author-pic"&gt;

          &lt;a href="/ayanarshad02" class="crayons-avatar  crayons-avatar--l  "&gt;
            &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3817591%2F0fa1e4d1-8be4-4467-b7f3-2fa3543e804b.png" alt="ayanarshad02 profile" class="crayons-avatar__image"&gt;
          &lt;/a&gt;
        &lt;/div&gt;
        &lt;div&gt;
          &lt;div&gt;
            &lt;a href="/ayanarshad02" class="crayons-story__secondary fw-medium m:hidden"&gt;
              Md Ayan Arshad
            &lt;/a&gt;
            &lt;div class="profile-preview-card relative mb-4 s:mb-0 fw-medium hidden m:inline-block"&gt;
              
                Md Ayan Arshad
                
              
              &lt;div id="story-author-preview-content-3627495" class="profile-preview-card__content crayons-dropdown branded-7 p-4 pt-0"&gt;
                &lt;div class="gap-4 grid"&gt;
                  &lt;div class="-mt-4"&gt;
                    &lt;a href="/ayanarshad02" class="flex"&gt;
                      &lt;span class="crayons-avatar crayons-avatar--xl mr-2 shrink-0"&gt;
                        &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3817591%2F0fa1e4d1-8be4-4467-b7f3-2fa3543e804b.png" class="crayons-avatar__image" alt=""&gt;
                      &lt;/span&gt;
                      &lt;span class="crayons-link crayons-subtitle-2 mt-5"&gt;Md Ayan Arshad&lt;/span&gt;
                    &lt;/a&gt;
                  &lt;/div&gt;
                  &lt;div class="print-hidden"&gt;
                    
                      Follow
                    
                  &lt;/div&gt;
                  &lt;div class="author-preview-metadata-container"&gt;&lt;/div&gt;
                &lt;/div&gt;
              &lt;/div&gt;
            &lt;/div&gt;

          &lt;/div&gt;
          &lt;a href="https://dev.to/ayanarshad02/i-increased-retrieval-from-top-5-to-top-20-my-answers-got-worse-3mke" class="crayons-story__tertiary fs-xs"&gt;&lt;time&gt;May 7&lt;/time&gt;&lt;span class="time-ago-indicator-initial-placeholder"&gt;&lt;/span&gt;&lt;/a&gt;
        &lt;/div&gt;
      &lt;/div&gt;

    &lt;/div&gt;

    &lt;div class="crayons-story__indention"&gt;
      &lt;h2 class="crayons-story__title crayons-story__title-full_post"&gt;
        &lt;a href="https://dev.to/ayanarshad02/i-increased-retrieval-from-top-5-to-top-20-my-answers-got-worse-3mke" id="article-link-3627495"&gt;
          I Increased Retrieval From Top-5 to Top-20. My Answers Got Worse
        &lt;/a&gt;
      &lt;/h2&gt;
        &lt;div class="crayons-story__tags"&gt;
            &lt;a class="crayons-tag crayons-tag--filled  " href="/t/discuss"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;discuss&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/ai"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;ai&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/programming"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;programming&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/machinelearning"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;machinelearning&lt;/a&gt;
        &lt;/div&gt;
      &lt;div class="crayons-story__bottom"&gt;
        &lt;div class="crayons-story__details"&gt;
          &lt;a href="https://dev.to/ayanarshad02/i-increased-retrieval-from-top-5-to-top-20-my-answers-got-worse-3mke" class="crayons-btn crayons-btn--s crayons-btn--ghost crayons-btn--icon-left"&gt;
            &lt;div class="multiple_reactions_aggregate"&gt;
              &lt;span class="multiple_reactions_icons_container"&gt;
                  &lt;span class="crayons_icon_container"&gt;
                    &lt;img src="https://assets.dev.to/assets/sparkle-heart-5f9bee3767e18deb1bb725290cb151c25234768a0e9a2bd39370c382d02920cf.svg" width="18" height="18"&gt;
                  &lt;/span&gt;
              &lt;/span&gt;
              &lt;span class="aggregate_reactions_counter"&gt;1&lt;span class="hidden s:inline"&gt;&amp;nbsp;reaction&lt;/span&gt;&lt;/span&gt;
            &lt;/div&gt;
          &lt;/a&gt;
            &lt;a href="https://dev.to/ayanarshad02/i-increased-retrieval-from-top-5-to-top-20-my-answers-got-worse-3mke#comments" class="crayons-btn crayons-btn--s crayons-btn--ghost crayons-btn--icon-left flex items-center"&gt;
              

              2&lt;span class="hidden s:inline"&gt;&amp;nbsp;comments&lt;/span&gt;
            &lt;/a&gt;
        &lt;/div&gt;
        &lt;div class="crayons-story__save"&gt;
          &lt;small class="crayons-story__tertiary fs-xs mr-2"&gt;
            6 min read
          &lt;/small&gt;
            
              &lt;span class="bm-initial crayons-icon c-btn__icon"&gt;
                

              &lt;/span&gt;
              &lt;span class="bm-success crayons-icon c-btn__icon"&gt;
                

              &lt;/span&gt;
            
        &lt;/div&gt;
      &lt;/div&gt;
    &lt;/div&gt;
  &lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;


</description>
    </item>
    <item>
      <title>I Increased Retrieval From Top-5 to Top-20. My Answers Got Worse</title>
      <dc:creator>Md Ayan Arshad</dc:creator>
      <pubDate>Thu, 07 May 2026 13:53:07 +0000</pubDate>
      <link>https://dev.to/ayanarshad02/i-increased-retrieval-from-top-5-to-top-20-my-answers-got-worse-3mke</link>
      <guid>https://dev.to/ayanarshad02/i-increased-retrieval-from-top-5-to-top-20-my-answers-got-worse-3mke</guid>
      <description>&lt;p&gt;The standard advice for improving RAG retrieval quality is: retrieve more candidates, then filter down. Bigger pool, better reranker, better answers. I followed that advice in &lt;a href="https://github.com/AyanArshad02/kapa-inspired-rag-mcp" rel="noopener noreferrer"&gt;my RAG System&lt;/a&gt;. On PDFs, going from top-5 to top-20 made my RAGAS scores drop. The answers got worse, not better.&lt;/p&gt;

&lt;p&gt;Here's what actually happened and the experiment design that explained it.&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;PDFs&lt;/strong&gt; (40 QA pairs, 5 technical documents):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Condition&lt;/th&gt;
&lt;th&gt;RAGAS SUM&lt;/th&gt;
&lt;th&gt;Context Precision&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;top-5, no reranker (baseline)&lt;/td&gt;
&lt;td&gt;3.4330&lt;/td&gt;
&lt;td&gt;0.8102&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;top-20, no reranker&lt;/td&gt;
&lt;td&gt;3.4051 ↓&lt;/td&gt;
&lt;td&gt;0.8118&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;top-20 → Cohere rerank → top-5&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;3.4843&lt;/strong&gt; ↑&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.8368&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;GitHub code&lt;/strong&gt; (50 QA pairs, &lt;code&gt;encode/httpx&lt;/code&gt; repo):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Condition&lt;/th&gt;
&lt;th&gt;RAGAS SUM&lt;/th&gt;
&lt;th&gt;Context Precision&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;top-5, no reranker (baseline)&lt;/td&gt;
&lt;td&gt;3.5680&lt;/td&gt;
&lt;td&gt;0.7812&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;top-20, no reranker&lt;/td&gt;
&lt;td&gt;3.5766&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;0.7812&lt;/strong&gt; ← identical&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;top-20 → Cohere rerank → top-5&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;3.7079&lt;/strong&gt; ↑&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.9335&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On PDFs, more candidates without a quality filter made scores drop. On code, a 4x larger pool produced zero improvement in Context Precision i.e. 0.7812 versus 0.7812. Every gain in both cases came entirely from the reranker.&lt;/p&gt;

&lt;h2&gt;
  
  
  The standard advice
&lt;/h2&gt;

&lt;p&gt;Most RAG tutorials recommend something like: retrieve top-20 or top-50 candidates, then rerank to top-5. The reasoning is intuitive, a bigger retrieval pool gives the reranker more material to work with, so the final 5 chunks are better quality.&lt;/p&gt;

&lt;p&gt;That reasoning isn't wrong. But it hides an important assumption: the reranker is present. Without it, a bigger pool doesn't help. It actively hurts.&lt;/p&gt;

&lt;p&gt;To separate these two effects, I designed a 3 condition experiment. Most people only test "with reranker vs without reranker", that confounds pool size and reranking quality into one comparison. Breaking it into three conditions isolates what's actually causing the change.&lt;/p&gt;

&lt;h2&gt;
  
  
  The 3-condition experiment design
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Condition A: top-5,  no reranker    → baseline
Condition B: top-20, no reranker    → isolates pool size effect
Condition C: top-20 → Cohere → top-5 → isolates reranker contribution
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The logic:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;If C &amp;gt; B: the reranker is doing real work, not just benefiting from more candidates&lt;/li&gt;
&lt;li&gt;If B &amp;gt; A: a bigger pool helps even without reranking&lt;/li&gt;
&lt;li&gt;If B ≈ A or B &amp;lt; A: pool size doesn't matter and all improvement comes from the reranker&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Running this on two different data types produced two different failure modes. Both pointed at the same root cause.&lt;/p&gt;

&lt;h2&gt;
  
  
  Result 1 : PDFs
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Corpus:&lt;/strong&gt; 5 technical PDFs from FastAPI, Kubernetes, React, Stripe API reference, AWS overview along with 40 QA pairs&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Condition&lt;/th&gt;
&lt;th&gt;Faithfulness&lt;/th&gt;
&lt;th&gt;Ctx Precision&lt;/th&gt;
&lt;th&gt;Ctx Recall&lt;/th&gt;
&lt;th&gt;RAGAS SUM&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;top-5, no reranker&lt;/td&gt;
&lt;td&gt;0.9137&lt;/td&gt;
&lt;td&gt;0.8102&lt;/td&gt;
&lt;td&gt;0.8917&lt;/td&gt;
&lt;td&gt;3.4330&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;top-20, no reranker&lt;/td&gt;
&lt;td&gt;0.9004&lt;/td&gt;
&lt;td&gt;0.8118&lt;/td&gt;
&lt;td&gt;0.8750&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;3.4051&lt;/strong&gt; ↓&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;top-20 → Cohere → top-5&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.9267&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.8368&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.8929&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;3.4843&lt;/strong&gt; ↑&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Condition B scored lower than Condition A on every metric except Context Precision, where it gained 0.0016 and its statistically meaningless. Overall RAGAS SUM dropped from 3.4330 to 3.4051.&lt;/p&gt;

&lt;p&gt;More candidates made the answers worse.&lt;/p&gt;

&lt;p&gt;The reranker (Condition C) recovered the loss and added on top of it: SUM 3.4843, Context Precision 0.8368. The difference between C and B is entirely the reranker's contribution.&lt;/p&gt;

&lt;h2&gt;
  
  
  Result 2 : GitHub code
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Corpus:&lt;/strong&gt; &lt;code&gt;encode/httpx&lt;/code&gt; repository and 90 files, 50 QA pairs on function behavior and parameters. Full experiment code and eval sets are in the &lt;a href="https://github.com/AyanArshad02/kapa-inspired-rag-mcp" rel="noopener noreferrer"&gt;repo&lt;/a&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Condition&lt;/th&gt;
&lt;th&gt;Ctx Precision&lt;/th&gt;
&lt;th&gt;Ctx Recall&lt;/th&gt;
&lt;th&gt;RAGAS SUM&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;top-5, no reranker&lt;/td&gt;
&lt;td&gt;0.7812&lt;/td&gt;
&lt;td&gt;0.9700&lt;/td&gt;
&lt;td&gt;3.5680&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;top-20, no reranker&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.7812&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0.9700&lt;/td&gt;
&lt;td&gt;3.5766&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;top-20 → Cohere → top-5&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.9335&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0.9300&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;3.7079&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Condition B versus Condition A: Context Precision 0.7812 versus 0.7812. Identical. A 4x larger retrieval pool produced zero improvement in precision.&lt;/p&gt;

&lt;p&gt;Then the reranker: Context Precision jumps from 0.7812 to 0.9335. That's +0.1523, the largest precision gain in any experiment across this entire project. RAGAS SUM 3.7079 is the highest score in the project. The PDF best was 3.4843.&lt;/p&gt;

&lt;p&gt;One tradeoff worth naming: Context Recall dropped slightly from 0.9700 to 0.9300 when the reranker was added. The reranker filters aggressively for relevance, occasionally it discards a chunk that contained useful information but didn't score highest on the query. For most QA use cases, a +0.1523 precision gain at the cost of -0.0400 recall is clearly the right tradeoff. But it's real, and worth monitoring if recall matters more than precision for your use case.&lt;/p&gt;

&lt;p&gt;Every point of the precision improvement came from the reranker, not from the pool size.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this happens
&lt;/h2&gt;

&lt;p&gt;Without a reranker, the top-k selection is purely based on embedding similarity. The embedding model retrieves the 20 chunks whose vectors are closest to the query vector. In the top 5, those are the 5 closest. In the top 20, you get those same 5 plus 15 more, which are further away in embedding space and increasingly likely to be noise.&lt;/p&gt;

&lt;p&gt;Those 15 extra chunks go directly into the LLM's context window. The LLM sees 20 chunks instead of 5. The signal-to-noise ratio drops. The answers get worse.&lt;/p&gt;

&lt;p&gt;The reranker changes the game because it operates on a completely different signal. Cohere's reranker doesn't use vector proximity — it reads the query and each chunk as text, then scores relevance directly. It can distinguish between a chunk that contains the query's keywords but doesn't answer the question, and a chunk that answers the question using different words. Embedding similarity can't do that.&lt;/p&gt;

&lt;p&gt;So the reranker takes the noisy top-20 pool and discards 15 chunks. The 5 it keeps are genuinely relevant, not just vectorially close. That's why Context Precision jumped from 0.7812 to 0.9335 on code and why adding more candidates without the reranker did nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The "reranker does real work" proof
&lt;/h2&gt;

&lt;p&gt;The 3 condition design specifically tests this.&lt;/p&gt;

&lt;p&gt;If all the improvement in Condition C came from the larger pool rather than the reranker, then Condition B (same pool, no reranker) would show similar gains. It didn't, on code, B and A were identical. On PDFs, B was worse than A.&lt;/p&gt;

&lt;p&gt;Every gain in Condition C came from the reranker acting on the larger pool. The pool size is not the lever. The reranker is.&lt;/p&gt;

&lt;p&gt;This matters practically. A common optimization people reach for is "increase k." It's a one line config change. But the data shows it has no effect without a reranker, and can actively hurt. The right lever is adding a reranker, not increasing k.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I learned
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Increasing retrieval candidates without a reranker adds noise, not signal, on PDFs, top-20 without a reranker scored lower than top-5 on every metric&lt;/li&gt;
&lt;li&gt;On code, expanding from top-5 to top-20 produced 0.0000 improvement in Context Precision, the pool size was genuinely irrelevant&lt;/li&gt;
&lt;li&gt;The 3 condition design (top-5 / top-20 / top-20+rerank) is the correct way to test this, "with vs without reranker" conflates two separate effects&lt;/li&gt;
&lt;li&gt;The reranker's advantage is operating on text, not vectors, it catches semantic relevance that embedding similarity misses&lt;/li&gt;
&lt;li&gt;+0.1523 Context Precision on code is the largest single-component gain in this project, one API call, one reranker, that result&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The practical takeaway
&lt;/h2&gt;

&lt;p&gt;If you're trying to improve RAG answer quality, don't reach for a larger k first.&lt;/p&gt;

&lt;p&gt;Add a reranker. Then increase k if you want to give it more to work with.&lt;/p&gt;

&lt;p&gt;Increasing k without a reranker gives the LLM more context to get confused by. With a reranker, a larger pool means the right chunks are more likely to be in the candidate set before filtering. The order matters.&lt;/p&gt;

&lt;p&gt;A top-20 retrieve → Cohere rerank → top-5 pipeline consistently outperformed both top-5 (baseline) and top-20 without reranking across two separate data types and 90 total QA pairs. The pattern is stable.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Part of an ongoing series on building and evaluating a production RAG system.&lt;/em&gt;&lt;br&gt;
&lt;em&gt;Full code in GitHub : &lt;a href="https://github.com/AyanArshad02/kapa-inspired-rag-mcp" rel="noopener noreferrer"&gt;Reverse Engineering YC Startup&lt;/a&gt;&lt;/em&gt;&lt;br&gt;
&lt;em&gt;Previous post: &lt;a href="https://dev.to/ayanarshad02/i-tested-chunking-on-docs-pdfs-and-code-the-winner-changed-every-time-1lof"&gt;I Tested Chunking on Docs, PDFs, and Code. The Winner Changed Every Time.&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>machinelearning</category>
      <category>discuss</category>
    </item>
    <item>
      <title>I Tested Chunking on Docs, PDFs, and Code. The Winner Changed Every Time.</title>
      <dc:creator>Md Ayan Arshad</dc:creator>
      <pubDate>Mon, 04 May 2026 16:32:41 +0000</pubDate>
      <link>https://dev.to/ayanarshad02/i-tested-chunking-on-docs-pdfs-and-code-the-winner-changed-every-time-1lof</link>
      <guid>https://dev.to/ayanarshad02/i-tested-chunking-on-docs-pdfs-and-code-the-winner-changed-every-time-1lof</guid>
      <description>&lt;p&gt;I assumed chunking was a solved problem. Pick a text splitter, set 512 tokens, add some overlap, move on. After running structured experiments across three different data types, that assumption collapsed. The best chunker for markdown documentation actively hurt performance on code. The winner changed completely depending on what I was chunking.&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Data type&lt;/th&gt;
&lt;th&gt;Winner&lt;/th&gt;
&lt;th&gt;Headline metric&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Markdown docs&lt;/td&gt;
&lt;td&gt;HeadingAwareChunker&lt;/td&gt;
&lt;td&gt;MRR 0.755 vs SlidingWindow 0.687&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PDFs&lt;/td&gt;
&lt;td&gt;RecursiveChar (512 tok)&lt;/td&gt;
&lt;td&gt;Context Recall 0.9250, RAGAS SUM 3.4249&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GitHub code&lt;/td&gt;
&lt;td&gt;CodeBlockAwareChunker&lt;/td&gt;
&lt;td&gt;RAGAS SUM 3.5680 — highest across all experiments&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;RecursiveChar won on PDFs. The same chunker scored 0.5690 Context Precision on code, roughly half the retrieved chunks were irrelevant. There is no universal best chunker. The data type decides.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I was building
&lt;/h2&gt;

&lt;p&gt;A RAG system that ingests documentation sites, PDFs, and GitHub repositories for multiple tenants, then answers developer questions with citations. Before embedding anything, I had to decide how to chunk each source type.&lt;/p&gt;

&lt;p&gt;The standard advice is "use a recursive text splitter." Every tutorial does this. But markdown docs have headings, PDFs have paragraphs, code has functions. A function is a complete semantic unit, splitting it at token 256 and you've lost the return type, the error handling, the docstring. None of that is recoverable at query time&lt;/p&gt;

&lt;p&gt;So I ran experiments, one variable changed per experiment: the chunker&lt;/p&gt;

&lt;p&gt;The embedding model, retrieval method, reranker, LLM, and eval set stayed fixed.&lt;br&gt;
RAGAS scored every pipeline on the same frozen question set.&lt;/p&gt;

&lt;p&gt;3 data types, 3 experiments and here's what happened.&lt;/p&gt;

&lt;p&gt;The full implementation, experiment notebooks, and eval sets are on &lt;a href="https://github.com/AyanArshad02/kapa-inspired-rag-mcp" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F18ag03gcgh5ra7wfbk26.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F18ag03gcgh5ra7wfbk26.png" alt=" " width="800" height="469"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Experiment 1: Documentation (.md / .mdx)
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Corpus:&lt;/strong&gt; FastAPI and Supabase documentation, 78 QA pairs generated by GPT-4o, frozen after generation&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Chunkers tested:&lt;/strong&gt; HeadingAwareChunker (HAC), SlidingWindow-128, RecursiveChar, SemanticBlock&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Key metric:&lt;/strong&gt; MRR (Mean Reciprocal Rank), recall@5 tells you if the answer is somewhere in the top 5. MRR tells you if it's at rank 1,  whether the right chunk comes first, not just eventually&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Chunker&lt;/th&gt;
&lt;th&gt;MRR (no reranker)&lt;/th&gt;
&lt;th&gt;Chunks produced&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;HeadingAwareChunker&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.755&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;127&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SlidingWindow-128&lt;/td&gt;
&lt;td&gt;0.687&lt;/td&gt;
&lt;td&gt;259&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;HAC produced the same Recall@5 as SlidingWindow (~0.82) but with significantly better MRR. The right answer appeared at rank 1 more often. And HAC did it with 127 chunks versus SlidingWindow's 259, half the chunks, better ranking, cheaper retrieval&lt;/p&gt;

&lt;p&gt;Why? Markdown documentation is already structured by headings. Each section covers one concept, one API endpoint, one configuration option. HAC splits exactly at those heading boundaries. SlidingWindow ignores them entirely, it cuts at token count, which means a chunk might start halfway through one concept and end halfway through the next.&lt;/p&gt;

&lt;p&gt;The embedding model then has to encode a chunk that mixes two ideas. The resulting vector is somewhere between them, and retrieval becomes imprecise.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Winner: HeadingAwareChunker.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Experiment 2: PDFs
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Corpus:&lt;/strong&gt; 5 technical PDFs from FastAPI concepts, Kubernetes architecture, React patterns, Stripe API reference, AWS overview along with 40 QA pairs&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Chunkers tested:&lt;/strong&gt; SlidingWindow-128, SemanticBlock, RecursiveChar (512 tokens, 50 overlap). HeadingAwareChunker was not included here, &lt;code&gt;pymupdf4llm&lt;/code&gt; extracts PDFs to Markdown, but the heading hierarchy in PDFs is inconsistent across documents. Font-size-based heading detection is fragile enough that HAC's boundaries would be unreliable. The experiment focused on chunkers that work on paragraph level structure, which is what the extraction reliably produces&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Chunker&lt;/th&gt;
&lt;th&gt;Context Recall&lt;/th&gt;
&lt;th&gt;RAGAS SUM&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;RecursiveChar&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.9250&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;3.4249&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SlidingWindow-128&lt;/td&gt;
&lt;td&gt;0.8750&lt;/td&gt;
&lt;td&gt;3.3691&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SemanticBlock&lt;/td&gt;
&lt;td&gt;0.8167&lt;/td&gt;
&lt;td&gt;3.2627&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;RecursiveChar won by a clear margin. Context Recall 0.9250 versus SlidingWindow's 0.8750.&lt;/p&gt;

&lt;p&gt;The reason is specific to how I extracted the PDFs. I used &lt;code&gt;pymupdf4llm&lt;/code&gt;, which converts PDFs to Markdown. The output is clean paragraphs with heading markers. RecursiveChar's default split points, double newlines, single newlines and it aligns naturally with those paragraph boundaries. It didn't need to classify blocks or detect headings. The structure was already there; RC just respected it.&lt;/p&gt;

&lt;p&gt;SemanticBlock failed on the Stripe API PDF. That document's navigation sidebar produced 12-token noise chunks, fragment after fragment of menu items. Those wasted retrieval slots on every single query.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Winner: RecursiveChar.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Note what just happened: HAC won on docs, RC won on PDFs. Two different data types, two different winners and the experiments are only half done&lt;/p&gt;

&lt;h2&gt;
  
  
  Experiment 3: GitHub code
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Corpus:&lt;/strong&gt; &lt;code&gt;encode/httpx&lt;/code&gt; repository having 90 files (60 Python, 29 Markdown, 1 text). 50 QA pairs focused on function behavior, parameters, and return values&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Chunkers tested:&lt;/strong&gt; CodeBlockAwareChunker (CBAC), RecursiveChar, SlidingWindow-128&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Chunker&lt;/th&gt;
&lt;th&gt;Ctx Precision&lt;/th&gt;
&lt;th&gt;Ctx Recall&lt;/th&gt;
&lt;th&gt;RAGAS SUM&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;CodeBlockAwareChunker&lt;/td&gt;
&lt;td&gt;0.7812&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.9700&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;3.5680&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SlidingWindow-128&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.8278&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0.9150&lt;/td&gt;
&lt;td&gt;3.4957&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RecursiveChar&lt;/td&gt;
&lt;td&gt;0.5690&lt;/td&gt;
&lt;td&gt;0.9400&lt;/td&gt;
&lt;td&gt;3.2856&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;RecursiveChar scored 0.5690 on Context Precision, that means roughly half of the retrieved chunks were irrelevant to the question. The same chunker that won on PDFs failed on code.&lt;/p&gt;

&lt;p&gt;The failure mode is direct. Python code is full of blank lines between a function's docstring and its body, between logical sections inside a method, between a guard clause and the main logic. RecursiveChar splits at blank lines. So it routinely bundled two or three unrelated functions into a single chunk, averaging 457 tokens. When someone asks "what does &lt;code&gt;Client.send()&lt;/code&gt; return," the retrieved chunk contains &lt;code&gt;send()&lt;/code&gt; plus &lt;code&gt;get()&lt;/code&gt; plus the &lt;code&gt;__init__&lt;/code&gt; method. Everything but a focused answer.&lt;/p&gt;

&lt;p&gt;CBAC doesn't use blank lines. For Python files, it uses the &lt;code&gt;ast&lt;/code&gt; module, it finds the exact byte offset of every function and class definition in the syntax tree, then extracts each one as a separate chunk. Zero false splits. The average chunk was 120 tokens, one complete function.&lt;/p&gt;

&lt;p&gt;SlidingWindow 128 had the best Context Precision (0.8278), small windows avoid the bundling problem. But it split functions mid-body. A function's return value might land in the next window. That killed Recall: 0.9150 versus CBAC's 0.9700.&lt;/p&gt;

&lt;p&gt;CBAC with a full reranker pipeline achieved RAGAS SUM 3.7079, the highest score across all experiments in this project and the PDF best was 3.4843&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Winner: CodeBlockAwareChunker.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the results differ and why they shouldn't surprise you
&lt;/h2&gt;

&lt;p&gt;Each experiment picked a different chunker, but every result points at the same question: what is the natural semantic unit of this data?&lt;/p&gt;

&lt;p&gt;For markdown documentation, it's the section under a heading. That's a discrete concept, authored that way intentionally.&lt;/p&gt;

&lt;p&gt;For PDFs extracted to Markdown, it's the paragraph. The extraction tool already produces those boundaries. The chunker just has to respect them.&lt;/p&gt;

&lt;p&gt;For code, it's the function or class. A function is the smallest unit of behavior that makes sense alone. Split it and the chunk becomes meaningless without the surrounding context.&lt;/p&gt;

&lt;p&gt;Text splitters, recursive or sliding window, don't know any of this. They operate on character counts, token counts, or blank lines. None of those correspond to semantic boundaries in code. That's the root cause of RecursiveChar's 0.5690 Context Precision. It wasn't a hyperparameter problem. It was a conceptual mismatch.&lt;/p&gt;

&lt;p&gt;There's also a second effect worth naming: chunk count matters. HAC's 127 chunks versus SlidingWindow's 259 on the same corpus is not a coincidence. Fewer chunks means fewer candidates for noise to enter the retrieval pool. The embedding space is less diluted and rank 1 is cleaner&lt;/p&gt;

&lt;h2&gt;
  
  
  What I learned
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The optimal chunker is determined by the data type, not by chunk size or overlap settings&lt;/li&gt;
&lt;li&gt;RecursiveChunker's blank-line heuristic is a real liability for code, 0.5690 Context Precision proves it&lt;/li&gt;
&lt;li&gt;Smaller average chunks (120 tokens) outperformed larger ones (457 tokens) on code by a significant margin, chunk size is a symptom, not a cause&lt;/li&gt;
&lt;li&gt;Visual inspection of actual chunks before running RAGAS catches structural bugs that aggregate scores smooth over, I caught CBAC producing 8KB chunks on Go files before the experiment ran&lt;/li&gt;
&lt;li&gt;Freezing the eval set before the first experiment is non negotiable because regenerating it mid experiment would invalidate every comparison&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The practical takeaway
&lt;/h2&gt;

&lt;p&gt;There is no universal best chunker&lt;/p&gt;

&lt;p&gt;For markdown documentation: split at heading boundaries&lt;br&gt;
For PDFs: convert to Markdown first, then split at paragraph boundaries&lt;br&gt;
For code: use an AST parser&lt;/p&gt;

&lt;p&gt;A generic 512-token splitter will technically work on all three. It will not be optimal on any of them. And on code specifically, the degradation is not marginal, it's a near-halving of retrieval precision.&lt;/p&gt;

&lt;p&gt;Pick the chunker that matches the semantic structure of the data, not the one that's easiest to configure.&lt;/p&gt;

&lt;p&gt;The harder version of this problem is mixed content, a PDF with embedded code blocks, a GitHub repo where half the files are Python and half are Markdown. Each file type still needs its own chunking strategy, which means the chunker has to detect content type at the file level and route accordingly. That's what the connector layer in this project handles, but it's a separate problem worth its own post.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;I'm building a &lt;a href="https://github.com/AyanArshad02/kapa-inspired-rag-mcp" rel="noopener noreferrer"&gt;production RAG system&lt;/a&gt; that ingests multiple source types with per-source-type chunking strategies. Future posts cover the reranker experiments, eval methodology, and the CI pipeline I built around RAGAS scores.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>discuss</category>
      <category>programming</category>
      <category>datascience</category>
    </item>
    <item>
      <title>5 Critical Failures We Hit Shipping a Multi-Tenant RAG Chatbot to 500+ Enterprises</title>
      <dc:creator>Md Ayan Arshad</dc:creator>
      <pubDate>Sat, 04 Apr 2026 06:26:56 +0000</pubDate>
      <link>https://dev.to/ayanarshad02/we-shipped-a-rag-chatbot-to-500-enterprise-tenants-heres-what-actually-broke-first-1jia</link>
      <guid>https://dev.to/ayanarshad02/we-shipped-a-rag-chatbot-to-500-enterprise-tenants-heres-what-actually-broke-first-1jia</guid>
      <description>&lt;p&gt;Our first enterprise tenant onboarded on a Monday.&lt;/p&gt;

&lt;p&gt;By Wednesday, 30% of their documents had been silently indexed as empty strings. No error. No exception. The chatbot just said "I don't have enough information", confidently, every time.&lt;/p&gt;

&lt;p&gt;That was Failure #1. There were four more.&lt;/p&gt;

&lt;p&gt;Here's the honest account of shipping a multi-tenant RAG chatbot to 500+ enterprise clients — what broke, in what order, and what we should have caught earlier.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fsulg56rhevv4oloo38vj.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fsulg56rhevv4oloo38vj.png" alt=" " width="800" height="719"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The System We Built
&lt;/h2&gt;

&lt;p&gt;Before the failures, the context.&lt;/p&gt;

&lt;p&gt;We built a RAG chatbot for enterprise warehouse management. Each tenant had their own isolated knowledge base — SOPs, compliance documents, operational guides. Users queried only their tenant's data. Scale target: ~25,000 queries per day at full rollout.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Indexing pipeline:&lt;/strong&gt;&lt;br&gt;
&lt;code&gt;Document Upload → Type Detection → Preprocessing → Chunking → Embedding → Pinecone&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Query pipeline:&lt;/strong&gt;&lt;br&gt;
&lt;code&gt;User Query → Cache Check → Query Rewrite → Hybrid Search (BM25 + Vector) → RRF Fusion → Reranker → LLM → Response&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;Two pipelines in the design. One EC2 fleet in reality, which became Failure #4.&lt;/p&gt;

&lt;p&gt;Indexing consumed from SQS. Query API sat behind an ALB. One Pinecone namespace per tenant, every query scoped to the authenticated tenant's namespace before touching the vector DB.&lt;/p&gt;

&lt;p&gt;The architecture decisions were mostly right.&lt;/p&gt;

&lt;p&gt;What broke was the assumptions underneath them.&lt;/p&gt;
&lt;h2&gt;
  
  
  Failure #1: The PDF Preprocessing Assumption (Week 1)
&lt;/h2&gt;

&lt;p&gt;We assumed all enterprise documents were text-based PDFs.&lt;/p&gt;

&lt;p&gt;They weren't.&lt;/p&gt;

&lt;p&gt;About 30% of what tenants uploaded were scanned PDFs, images of physical pages, no text layer. When PyMuPDF opened these files, it returned empty strings. We embedded empty strings. We indexed empty chunks. No error. No exception. Just silent failure.&lt;/p&gt;

&lt;p&gt;Users asked questions. Retrieval returned nothing relevant. The LLM said "I don't have enough information." Users assumed the chatbot was broken. They were right, just not for the reason they thought.&lt;/p&gt;

&lt;p&gt;The fix: A preprocessing gate that checks average characters per page. If avg_chars_per_page &amp;lt; 100, no text layer exists, trigger OCR via AWS Textract before chunking. We also added an admin-facing flag marking documents as "pending OCR" so tenants know their document is processing, not lost.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The lesson:&lt;/strong&gt; Never assume your input format. Garbage input produces zero output in RAG. Preprocessing is the most boring part of the pipeline and the most catastrophic to skip.&lt;/p&gt;
&lt;h2&gt;
  
  
  Failure #2: Headers, Footers, and the Chunk Contamination Problem
&lt;/h2&gt;

&lt;p&gt;Even for text-based PDFs, every chunk was contaminated.&lt;/p&gt;

&lt;p&gt;Enterprise documents have headers and footers on every page. "Softeon WMS User Guide — Confidential — Page 14 of 203." When you chunk a 200-page document into 512-token pieces, that text bleeds into hundreds of chunks.&lt;/p&gt;

&lt;p&gt;The retrieval impact was subtle but real. Queries about "confidential" topics surfaced chunks with "Confidential" in the footer, not because the content was relevant, but because BM25 was matching on that exact term. Relevance scores were quietly polluted.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The fix:&lt;/strong&gt; A stripping step before chunking. Text appearing in the top 5% and bottom 5% of every page gets flagged and removed. We also converted tables to markdown before chunking, a raw table extracted as "Product Price Refund Laptop 999 30 days" is useless for retrieval. The same table as structured markdown is self-contained and semantically meaningful.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The lesson:&lt;/strong&gt; Most RAG tutorials skip directly to chunking size debates — 256 vs 512 tokens. They assume clean input. Real enterprise documents are not clean.&lt;/p&gt;
&lt;h2&gt;
  
  
  Failure #3: The Parallel Pipeline Was Actually Sequential
&lt;/h2&gt;

&lt;p&gt;We ran BM25 and vector search in what we thought was parallel.&lt;/p&gt;

&lt;p&gt;It wasn't.&lt;/p&gt;

&lt;p&gt;The original implementation called BM25, waited for the result, then called Pinecone. "Parallel" on the architecture diagram. Sequential in the code. At p50 this cost us ~200ms we couldn't afford.&lt;/p&gt;

&lt;p&gt;The fix is one line:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Wrong — sequential
&lt;/span&gt;&lt;span class="n"&gt;bm25_results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;bm25_search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;vector_results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;pinecone_search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Right — parallel
&lt;/span&gt;&lt;span class="n"&gt;bm25_results&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;vector_results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;gather&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nf"&gt;bm25_search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="nf"&gt;pinecone_search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Latency becomes the max of the two, not the sum. Dropped our p95 by ~180ms.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The lesson:&lt;/strong&gt; "Parallel" on a diagram and "parallel" in code are different things. Profile your pipeline stage by stage. The bottleneck is always somewhere surprising.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure #4: One Tenant's Upload Degraded Everyone's Query Latency
&lt;/h2&gt;

&lt;p&gt;This one took a week to diagnose.&lt;/p&gt;

&lt;p&gt;We noticed periodic p99 spikes, not consistent, not tied to query volume. Random, unpredictable.&lt;/p&gt;

&lt;p&gt;The cause: our indexing pipeline and query pipeline were on the same EC2 instances.&lt;/p&gt;

&lt;p&gt;When a large tenant uploaded 500 documents, the embedding loop hammered the instance CPU. Live users querying on the same instance saw response time jump from 800ms to 6+ seconds. The indexer and query service were invisible to each other in code — but very visible to each other on the metal.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The fix:&lt;/strong&gt; Complete infrastructure separation. Indexing workers on a dedicated EC2 fleet, completely outside the ALB. The query fleet has no knowledge that indexing is happening. A document upload spike now has zero effect on query latency for any tenant. The SQS queue buffers upload bursts and feeds indexing workers at a controlled pace.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The lesson:&lt;/strong&gt; Load isolation is not just an architectural principle. It's a user experience decision. Enterprise tenants don't care about your architecture, they care that the chatbot was slow when they needed it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure #5: The Namespace Isolation Gap We Almost Missed
&lt;/h2&gt;

&lt;p&gt;Multi-tenant isolation in Pinecone is handled by namespaces. One namespace per tenant. Every write tags it. Every read is scoped to it.&lt;/p&gt;

&lt;p&gt;What we almost shipped: namespace scoped at the request body level.&lt;/p&gt;

&lt;p&gt;A bad actor passing a forged tenant_id in the request body could scope the query to a different tenant's namespace. Subtle. Critical.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Wrong — trusting request body
&lt;/span&gt;&lt;span class="n"&gt;namespace&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tenant_id&lt;/span&gt;

&lt;span class="c1"&gt;# Right — trusting validated token only
&lt;/span&gt;&lt;span class="n"&gt;namespace&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token_context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tenant_id&lt;/span&gt;  &lt;span class="c1"&gt;# resolved from JWT at API layer
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;The fix:&lt;/strong&gt; Namespace resolved exclusively from the validated JWT token at the API layer. The request body's tenant_id is ignored entirely. By the time a request reaches the vector DB call, the namespace has already been locked to the authenticated tenant — it cannot be overridden.&lt;/p&gt;

&lt;p&gt;If we had shipped the original version, any authenticated user who knew another tenant's ID could have queried their private documents. In a WMS context serving enterprise clients, that's not a security incident, that's a contract termination and a legal conversation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The lesson:&lt;/strong&gt; Namespace isolation is not the same as security. Enforce tenant identity at authentication, not the application layer.&lt;/p&gt;

&lt;h2&gt;
  
  
  What We Still Haven't Built
&lt;/h2&gt;

&lt;p&gt;We don't have automated RAG evaluation in production.&lt;/p&gt;

&lt;p&gt;No RAGAS running continuously. No Precision@5 after every deployment. Human review by an internal QA team, representative queries, manual quality ratings. It works at current scale. It won't at full rollout.&lt;/p&gt;

&lt;p&gt;What I'd build next with two weeks:&lt;br&gt;
→ A golden evaluation set with 200 curated question-to-chunk pairs from real tenant queries. Your retrieval quality baseline.&lt;br&gt;
→ RAGAS Faithfulness in CI/CD runs on every deployment, blocks release if faithfulness drops more than 5% from baseline.&lt;br&gt;
→ Context Precision tracking, tells you if your reranker is actually earning its latency cost.&lt;/p&gt;

&lt;h2&gt;
  
  
  The One Thing That Mattered Most
&lt;/h2&gt;

&lt;p&gt;RAG systems fail at the edges of the pipeline, not the center.&lt;/p&gt;

&lt;p&gt;Most engineering effort goes into the center, embedding models, reranking algorithms, chunk sizes. The real production failures happen at the edges: what goes into the indexer, what happens when two workloads compete for the same compute, and where tenant identity gets resolved.&lt;/p&gt;

&lt;p&gt;What broke first in your RAG pipeline? Drop it in the comments. The failures nobody writes about are always the most useful. I'll compile the best ones into a follow-up post.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>agents</category>
      <category>programming</category>
      <category>security</category>
    </item>
    <item>
      <title>Why Our RAG System Was Silently Returning Wrong Answers — And How We Fixed It</title>
      <dc:creator>Md Ayan Arshad</dc:creator>
      <pubDate>Wed, 11 Mar 2026 00:19:49 +0000</pubDate>
      <link>https://dev.to/ayanarshad02/why-our-rag-system-was-silently-returning-wrong-answers-and-how-we-fixed-it-386g</link>
      <guid>https://dev.to/ayanarshad02/why-our-rag-system-was-silently-returning-wrong-answers-and-how-we-fixed-it-386g</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F9cgbkpcvxtjvlb2fylie.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F9cgbkpcvxtjvlb2fylie.webp" alt="Our RAG System Was Confidently Wrong"&gt;&lt;/a&gt;&lt;br&gt;
For 3 days, our RAG system was confident.&lt;/p&gt;

&lt;p&gt;Every query returned an answer. Response times were stable. No errors in the logs. By every operational metric, the system was working.&lt;/p&gt;

&lt;p&gt;Our RAGAS faithfulness score told a different story.&lt;/p&gt;

&lt;p&gt;It had dropped from 0.91 to 0.67 without a single code change.&lt;/p&gt;

&lt;p&gt;That meant roughly 1 in 3 responses was making claims our own retrieved context didn’t support. The system wasn’t crashing. It was hallucinating — silently, at scale, with complete confidence.&lt;/p&gt;

&lt;p&gt;Here is what happened.&lt;/p&gt;
&lt;h2&gt;
  
  
  The System When It Started Failing
&lt;/h2&gt;

&lt;p&gt;We were running a production RAG system serving enterprise clients across a large document corpus. Each client had their own isolated set of documents — product configuration files, setup guides, operational workflows — queried daily to answer operational questions.&lt;/p&gt;

&lt;p&gt;_The state of the system when the drift began:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;25K+ documents across corpus&lt;/li&gt;
&lt;li&gt;500+ enterprise tenants&lt;/li&gt;
&lt;li&gt;1 Pinecone namespace per tenant&lt;/li&gt;
&lt;li&gt;5 chunks retrieved per query (top-K)
_&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;Stack:&lt;/strong&gt; GPT-4 for generation, text-embedding-ada-002 for embeddings, Pinecone with one namespace per tenant, FastAPI on ECS. Isolation was strict — no cross-tenant reads, ever.&lt;/p&gt;

&lt;p&gt;_A NOTE on the namespace decision: Pinecone namespaces share the same index and billing unit, 500 namespaces cost the same as 1. We chose namespaces over metadata filtering (tenant_id filter on a single index) for one specific reason: Metadata filtering requires every query to carry the correct filter, and one bug means Tenant A can read Tenant B's data. For enterprise clients, that risk surface isn't acceptable. Namespaces make cross-tenant leakage structurally impossible at query time.&lt;/p&gt;

&lt;p&gt;Namespaces give us defense-in-depth isolation at the infrastructure layer rather than relying on application-level filtering._&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Monitoring:&lt;/strong&gt; API latency, error rates, cost dashboards.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Answer quality monitoring:&lt;/strong&gt; None.&lt;/p&gt;

&lt;p&gt;That was the bug. Not in the code. In the architecture.&lt;/p&gt;
&lt;h2&gt;
  
  
  What Changed, And Why We Didn’t See It
&lt;/h2&gt;

&lt;p&gt;Three days before we caught the drift, a large batch of new documents was ingested. Different document type than the existing corpus — denser, longer sentences, more domain-specific terminology. Same domain, different structure.&lt;/p&gt;

&lt;p&gt;The new documents changed the distribution of our Pinecone namespaces. Queries that had previously retrieved highly relevant chunks now retrieved chunks that were topically related but not directly answering the query.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cosine similarity scores:&lt;/strong&gt; 0.76, 0.79, 0.81. High enough to clear any threshold we’d set. &lt;code&gt;text-embedding-ada-002&lt;/code&gt; couldn't distinguish between "this chunk discusses this topic" and "this chunk contains the specific answer this query is asking for." Retrieval looked confident. The chunks were wrong.&lt;/p&gt;

&lt;p&gt;GPT-4 did what LLMs do when context is adjacent-but-imprecise: it filled the gaps with plausible-sounding claims not present in the retrieved text.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;RAGAS faithfulness:&lt;/strong&gt; 0.91 → 0.67.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Context Precision:&lt;/strong&gt; 0.84 → 0.61.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;We had no alert for either. We found out from a user.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The core failure:&lt;/strong&gt; we instrumented everything easy to measure — latency, throughput, cost, error rates — and nothing that measured correctness. We were flying blind on the one metric that determined whether the system was actually useful.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;One clarification on how we were running RAGAS.&lt;/strong&gt; We were not evaluating live traffic — that would be prohibitively expensive and slow. We maintained a golden evaluation set of ~300 representative queries with known-good answers and source chunks, curated when the system first launched. RAGAS ran against that set nightly, and on every ingestion event. The drop from 0.91 to 0.67 showed up the morning after the batch ingestion. We just had no alert configured to catch it.&lt;/p&gt;
&lt;h2&gt;
  
  
  What We Tried First — And Why It Wasn’t Enough
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;First instinct:&lt;/strong&gt; improve retrieval.&lt;/p&gt;

&lt;p&gt;We raised the similarity threshold from 0.70 to 0.78, rejecting chunks below that score. Retrieval precision improved. We also started returning no results for legitimate queries with unusual phrasing. Users got empty responses. That was worse.&lt;/p&gt;

&lt;p&gt;We increased top-K from 5 to 10. Slightly helped recall. Sent 2× the tokens into every LLM call, which compounded a cost problem already building at 500+ active tenants.&lt;/p&gt;

&lt;p&gt;Context Precision recovered to 0.78. Faithfulness only reached 0.81, still below our 0.85 target.&lt;/p&gt;

&lt;p&gt;The retrieval fixes were necessary. They were not sufficient. We needed a layer that caught the gap between what retrieval returned and what the LLM claimed. Retrieval improvement was treating the symptom. We needed to treat the cause.&lt;/p&gt;
&lt;h2&gt;
  
  
  The Real Fix: Grounding Validation as a First-Class Architecture Layer
&lt;/h2&gt;

&lt;p&gt;We added a grounding validation step that runs after every LLM response, before it’s returned to the user.&lt;/p&gt;

&lt;p&gt;The mechanism:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Extract the factual claims from the generated response using a structured extraction prompt. This step is imperfect, LLM-based claim extraction can miss or misinterpret implicit claims, so we treat it as a signal, not a definitive verdict. A claim is flagged as unsupported if no retrieved chunk scores above a similarity threshold for that claim.&lt;/li&gt;
&lt;li&gt;Score each claim against the retrieved chunks, classified as supported, unsupported, or contradicted.&lt;/li&gt;
&lt;li&gt;Flag any response where more than 15% of claims are unsupported or contradicted.&lt;/li&gt;
&lt;li&gt;Regenerate flagged responses with an explicit grounding instruction injected into the prompt.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;The regeneration prompt:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;“Your response must only make claims directly supported by the provided context. If the context does not contain the answer, say so explicitly. Do not infer or extrapolate.”&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Critical implementation detail:&lt;/strong&gt; The claim extraction and verification call runs against &lt;code&gt;gpt-4o-mini&lt;/code&gt;, not GPT-4. Running a full GPT-4 call for every response validation would double our inference cost and add 600–800ms of latency. With &lt;code&gt;gpt-4o-mini&lt;/code&gt;, the validation step adds approximately 180–220ms on average for a 3–5 sentence response. That number is model-dependent, it will be higher on a slower model and lower with a fine-tuned classifier.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;After deploying:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Before:  Faithfulness 0.67  |  Context Precision 0.61  |  31% unsupported claim rate
After:   Faithfulness 0.91  |  Context Precision 0.87  |  &amp;lt;4% unsupported claim rate
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F6lkl8ekyylllf9dylbt5.webp" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F6lkl8ekyylllf9dylbt5.webp" alt="Production RAG Architecture : 500+ Enterprise Clients"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The Core Decision — And When You’d Make It Differently
&lt;/h2&gt;

&lt;p&gt;We made an explicit trade-off: ~200ms of additional latency in exchange for verifiable answer quality.&lt;/p&gt;

&lt;p&gt;For our use case — enterprise clients making operational decisions based on the chatbot’s answers — that trade-off was not a discussion. The 200ms is noise. The trust cost of a wrong answer at enterprise scale is not.&lt;/p&gt;

&lt;p&gt;But this decision is not universal:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Constraint                            Decision          Rationale
──────────────────────────────────────────────────────────────────────────────
Consumer product, SLA &amp;lt; 500ms         Run async          Log failures, don't block.
                                                         Stakes are low, UX matters more.

Low-stakes (drafting, summarisation)  Skip it            User edits the output.
                                                         Grounding matters less.

High-volume, cost-sensitive           Sample 10%         Statistical signal at
                                                         1/10th the overhead.

Enterprise / regulated / high-stakes  Mandatory sync     A wrong answer has real
                                                         downstream consequences.

Multi-tenant, strict isolation        Mandatory + audit  Every response must be
                                                         traceable to a source chunk.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Principle:&lt;/strong&gt; grounding validation is always worth measuring. Whether to block on it synchronously depends on your SLA and the cost of a wrong answer in your domain.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Anti-Pattern: Checking Faithfulness After Generation Is the Wrong Architecture
&lt;/h2&gt;

&lt;p&gt;Here is the deeper architectural mistake this exposed.&lt;/p&gt;

&lt;p&gt;We were checking faithfulness after generation — as a post-hoc audit — rather than as a gating condition on the response pipeline. The audit told us something was wrong. It didn’t stop a wrong answer from being returned.&lt;/p&gt;

&lt;p&gt;The correct architecture treats grounding validation as a blocking step in the response pipeline, not an observability metric reviewed after the fact.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Wrong: Generate → Return → [async] Validate → Log failure → Review weekly
Right: Generate → Validate → [if flagged] Regenerate → Return → Log
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The async pattern gives you observability. It does not give you correctness. For any system where answer quality has downstream consequences, post-hoc monitoring is not a substitute for inline validation.&lt;/p&gt;

&lt;p&gt;We caught our failure because a user noticed. That should never be the detection mechanism for a production system serving enterprise clients.&lt;/p&gt;

&lt;h2&gt;
  
  
  What We Changed Permanently
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Quality monitoring is now first-class. RAGAS faithfulness and context precision scored against our golden evaluation set on every ingestion event and every deployment. Grafana alerts fire if either drops more than 10% from the established baseline. We do not run RAGAS on live traffic, it’s too slow and expensive at scale. The golden set gives us the signal we need.&lt;/li&gt;
&lt;li&gt;Document ingestion now triggers a quality gate. When new documents are ingested, we run the benchmark query set against the updated index before traffic is shifted. Faithfulness drops &amp;gt;5% → ingestion rolled back.&lt;/li&gt;
&lt;li&gt;Grounding validation is synchronous and non-configurable. The ~200ms cost is included in our SLA. Not optional for any enterprise-tier query.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The One-Line Takeaway
&lt;/h2&gt;

&lt;p&gt;Your RAG system will hallucinate. The question is whether you find out before your users do.&lt;/p&gt;

</description>
      <category>genai</category>
      <category>rag</category>
      <category>ai</category>
      <category>architecture</category>
    </item>
  </channel>
</rss>
