<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Angel</title>
    <description>The latest articles on DEV Community by Angel (@losiochico1).</description>
    <link>https://dev.to/losiochico1</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3259981%2F09493681-4f56-4015-9217-00e7558ba2cf.png</url>
      <title>DEV Community: Angel</title>
      <link>https://dev.to/losiochico1</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/losiochico1"/>
    <language>en</language>
    <item>
      <title>I built a RAG that refuses to answer — and measured when it should</title>
      <dc:creator>Angel</dc:creator>
      <pubDate>Thu, 19 Nov 2026 17:21:51 +0000</pubDate>
      <link>https://dev.to/losiochico1/i-built-a-rag-that-refuses-to-answer-and-measured-when-it-should-1ag</link>
      <guid>https://dev.to/losiochico1/i-built-a-rag-that-refuses-to-answer-and-measured-when-it-should-1ag</guid>
      <description>&lt;p&gt;Most RAG demos share a failure mode nobody demos: they always answer. On a&lt;br&gt;
corpus of medical evidence, that stops being a UX quirk and becomes a&lt;br&gt;
dangerous one.&lt;/p&gt;

&lt;p&gt;I built a dietary-supplements evidence RAG over 17,698 evidence-grade&lt;br&gt;
PubMed articles. It answers only when the retrieved studies support an&lt;br&gt;
answer, stamps every claim with the PMID it came from, and abstains at the&lt;br&gt;
corpus boundary instead of guessing.&lt;/p&gt;

&lt;p&gt;The part worth writing about isn't the pipeline — it's how I know the&lt;br&gt;
refusals are right.&lt;/p&gt;

&lt;h2&gt;
  
  
  The system, in one glance
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;gate → hybrid retrieve (BM25 + dense, RRF) → answer contract
     → deterministic verify + judge → one repair pass
     → answer with citations, or abstain
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Extractive-first: the default lane quotes study text instead of generating&lt;br&gt;
prose. Every run persists its full trace — gate decision, retrieved chunks,&lt;br&gt;
per-claim stamps, verdicts — at a public URL anyone can inspect.&lt;/p&gt;

&lt;h2&gt;
  
  
  The eval contract
&lt;/h2&gt;

&lt;p&gt;Two question sets, both committed to the repo.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;golden.jsonl&lt;/strong&gt; — 126 stratified questions. This is what I iterate against.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;heldout.jsonl&lt;/strong&gt; — 199 disjoint questions. This one only measures, never&lt;br&gt;
tunes. When golden and held-out disagree, I've overfit.&lt;/p&gt;

&lt;p&gt;The questions come from real sources (SuppLLMent effectiveness items,&lt;br&gt;
MedQuAD consumer questions), with one admission rule: a question enters the&lt;br&gt;
pool only if BM25 actually retrieves evidence for it. 4,026 made it in;&lt;br&gt;
1,543 were rejected — each with the reason kept in &lt;code&gt;rejected.jsonl&lt;/code&gt;. A&lt;br&gt;
question nobody can answer in the corpus isn't a test, it's noise.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the measurement found
&lt;/h2&gt;

&lt;p&gt;On the golden run, every abstention was correct — zero dangerous answers&lt;br&gt;
slipped through. That's the headline.&lt;/p&gt;

&lt;p&gt;The less comfortable finding: about 45% of &lt;em&gt;answerable&lt;/em&gt; rows abstained too.&lt;br&gt;
Partly the verifier being too strict, partly non-deterministic generation.&lt;br&gt;
So I added a rescue pass — when the system abstains but evidence sits at&lt;br&gt;
the top of the pack, it re-asks whether that evidence actually answers the&lt;br&gt;
question. Advice-decline rows went from 20% to 36% rescued; answerable rows&lt;br&gt;
stayed around 22%, because some of them legitimately lack coverage.&lt;/p&gt;

&lt;p&gt;That's the real shape of this work. Less "make the model better," more:&lt;br&gt;
measure a failure class, find the root cause, fix it, re-measure on a set&lt;br&gt;
you didn't tune against.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it, check it
&lt;/h2&gt;

&lt;p&gt;The live demo runs on my site — ask it about a supplement and watch it&lt;br&gt;
cite real studies, or watch it refuse. The repo and both eval sets are&lt;br&gt;
public.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Live:&lt;/strong&gt; &lt;a href="https://luisosio.dev" rel="noopener noreferrer"&gt;luisosio.dev&lt;/a&gt; (the supplements box, or the portfolio's own assistant — same "answer or abstain" architecture over the site's corpus)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Code + evals:&lt;/strong&gt; &lt;a href="https://github.com/LOsioChico/supplements-rag" rel="noopener noreferrer"&gt;github.com/LOsioChico/supplements-rag&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I'm an AI engineer in Colombia working remotely (UTC-5). If you're building&lt;br&gt;
systems where "I don't know" is a feature, I'd genuinely like to compare&lt;br&gt;
notes.&lt;/p&gt;

</description>
      <category>rag</category>
      <category>ai</category>
      <category>llm</category>
      <category>python</category>
    </item>
  </channel>
</rss>
