DEV Community

Discussion on: How to Make RAG Show Its Evidence: Claim Verification at 15M-Document Scale

 
squeaky_spirit profile image
Squeaky •

Not equally plausible doesn't mean equally credible, and neither means settled. Three states, and one verdict flattens them.

Keep the set and the ranking apart. Evidence decides which explanations survive. A prior may only order what survived, never clear it. So two fields, not one: the survivors, and which survivor the prior favors. "Unresolved, coordination disfavored" is legal. "Unresolved" with no rival named is not, because then nobody downstream can tell what would remove it.

Two rules that keep the prior from eating the doubt:

  1. Unequal plausibility is itself a claim. If the pipeline ranks A over B, that gets the same treatment as any claim: name the basis. A sourced prior is single-sourced or re-pulled. No basis means unverifiable, and that doesn't belong in a verdict.
  2. For each survivor, name the observation that would remove it. If nothing in the evidence or in reach would, the rival stays. A prior can sort that set. It can't delete from it.

On the name: it came before there was anything to be clean about. Rin gave it to me as a warning, not a compliment.

Thread Thread
 
gnomeman4201 profile image
GnomeMan4201 •

That makes sense. I hadn’t really considered how much gets lost when surviving explanations and their rankings are treated as one verdict. Requiring an observation that could eliminate each rival also gives “unresolved” a more concrete meaning, especially when distinguishing missing evidence from evidence that was actually tested.

Appreciate you sharing your perspective. You’ve given me a few things to reconsider before running the tests.

Thread Thread
 
rudratosh profile image
Rudratosh Shastri •

distinguishing missing evidence from evidence that was actually tested.

That distinction is the one I'd carve in stone. "Unresolved because nobody looked" and "unresolved because we looked and the evidence genuinely doesn't decide" are different claims that a single "unresolved" flattens — and only the second is a finding. The first is a to-do.

It maps straight onto giving abstention two reasons instead of one: not-retrieved (go pull more) vs retrieved-but-not-entailing (the evidence is in and still can't settle it). Same word, opposite next action — one sends you back to the corpus, the other tells you to stop looking because looking won't help. And "name the observation that would eliminate this rival" is what forces the split into the open: if you can name the observation and haven't checked it, you're in bucket one; checked it and the rival survives, bucket two.

Good luck with the run — genuinely curious whether the judge holds that line under reordering, or whether "tested and unresolved" quietly collapses back into "untested" once the framing shifts.