Most RAG demos share a failure mode nobody demos: they always answer. On a
corpus of medical evidence, that stops being a UX quirk and becomes a
dangerous one.
I built a dietary-supplements evidence RAG over 17,698 evidence-grade
PubMed articles. It answers only when the retrieved studies support an
answer, stamps every claim with the PMID it came from, and abstains at the
corpus boundary instead of guessing.
The part worth writing about isn't the pipeline — it's how I know the
refusals are right.
The system, in one glance
gate → hybrid retrieve (BM25 + dense, RRF) → answer contract
→ deterministic verify + judge → one repair pass
→ answer with citations, or abstain
Extractive-first: the default lane quotes study text instead of generating
prose. Every run persists its full trace — gate decision, retrieved chunks,
per-claim stamps, verdicts — at a public URL anyone can inspect.
The eval contract
Two question sets, both committed to the repo.
golden.jsonl — 126 stratified questions. This is what I iterate against.
heldout.jsonl — 199 disjoint questions. This one only measures, never
tunes. When golden and held-out disagree, I've overfit.
The questions come from real sources (SuppLLMent effectiveness items,
MedQuAD consumer questions), with one admission rule: a question enters the
pool only if BM25 actually retrieves evidence for it. 4,026 made it in;
1,543 were rejected — each with the reason kept in rejected.jsonl. A
question nobody can answer in the corpus isn't a test, it's noise.
What the measurement found
On the golden run, every abstention was correct — zero dangerous answers
slipped through. That's the headline.
The less comfortable finding: about 45% of answerable rows abstained too.
Partly the verifier being too strict, partly non-deterministic generation.
So I added a rescue pass — when the system abstains but evidence sits at
the top of the pack, it re-asks whether that evidence actually answers the
question. Advice-decline rows went from 20% to 36% rescued; answerable rows
stayed around 22%, because some of them legitimately lack coverage.
That's the real shape of this work. Less "make the model better," more:
measure a failure class, find the root cause, fix it, re-measure on a set
you didn't tune against.
Try it, check it
The live demo runs on my site — ask it about a supplement and watch it
cite real studies, or watch it refuse. The repo and both eval sets are
public.
- Live: luisosio.dev (the supplements box, or the portfolio's own assistant — same "answer or abstain" architecture over the site's corpus)
- Code + evals: github.com/LOsioChico/supplements-rag
I'm an AI engineer in Colombia working remotely (UTC-5). If you're building
systems where "I don't know" is a feature, I'd genuinely like to compare
notes.
Top comments (2)
The golden vs held-out split with a "measures, never tunes" rule is the part most eval write-ups skip, so this was good to see.
Curious about the ~45% over-abstain on answerable rows: did you check whether those cluster by question type (dosage vs efficacy, say) or by retrieval rank? If the supporting study sits at rank 1-2 and the verifier still rejects it, that usually means the claim-to-quote match is too literal. A paraphrase-tolerant check on just those rows might rescue more without moving the dangerous-answer rate, which you can confirm on held-out.
Also +1 on publishing the full trace per run. It makes a refusal something you can argue with instead of a mystery.
拒绝回答这个设计很有意思,尤其是在医疗证据这种场景下,宁可不答也不能瞎答。45% 的可回答问题也被拒绝这个数字很诚实,好奇救援步骤之后这个比例降到了多少。