Your retrieval demo is beautiful. You dropped a thousand PDFs into a vector database, wired up an embedding model, and now it answers questions wit...
For further actions, you may consider blocking this person and/or reporting abuse
I keep hitting a gap one level up in infrastructure attribution work. I structure conclusions as observation → evidence → unresolved inference, e.g. “These domains shared a backend during this window; ownership and coordination remain unestablished.”
Your Gate 3 should reject an explicit “these are coordinated” claim when the evidence only establishes shared infrastructure. The harder case is when every atomic claim is individually entailed (“A and B share an IP,” “both publish narrative X”), but the synthesis implies coordination without ever stating it. The leap lives between claims, not inside one.
Does the verifier have any view of the composed answer, or does that fall into the multi hop limitation you flagged?
Sharpest limitation anyone's raised, and you're right it's the one Gate 3 doesn't cover. Claim-level entailment certifies each atom and is blind to the composition — "A and B share an IP" and "both publish narrative X" can each be fully supported while the arrangement insinuates a "coordinated" that no chunk states. The implicature is in no claim, so nothing flags.
Two partial defenses, neither complete:
Does your pipeline enforce that unresolved-inference line, or is it a discipline you apply by hand?
That’s exactly the distinction I’m working through. Right now, the observation → evidence → unresolved inference structure is a methodological discipline, not something I’d claim the pipeline fully enforces.
I’ve been pretty stuck on this mindset lately, trying to keep my investigations as close to the raw evidence as possible without letting my own expectations influence the conclusions.
The tooling can preserve evidence, provenance, and explicit uncertainty, but that doesn’t automatically prevent a composed narrative from implying more than the individual observations establish.
Your point about making the unresolved inference a first class, verifiable output is particularly interesting. I think the harder test is whether the verifier can detect when a conclusion is implied by the arrangement, even when every individual statement passes entailment.
That suggests an interesting evaluation: hold the underlying evidence constant, vary only the framing and ordering of supported claims, and see whether the verifier recognizes the unsupported implication.
I’m curious how you’d approach detecting that kind of implication in practice. Would you treat it as an extension of entailment verification, or does it require a different kind of evaluation altogether?
Different evaluation, not an extension — and that's the uncomfortable part. Atomic entailment asks "does the evidence support this sentence," and the implied conclusion is, by construction, not any sentence in the output, so no amount of per-claim entailment reaches it. You have to extract the implied conclusion and make it an explicit claim, then run entailment on that against the union of cited evidence. Squeaky's framing in this thread is the cleanest version I've seen: for the implied conclusion, name the rival inference the same evidence licenses equally (shared backend ≠ shared owner) — if two incompatible conclusions come out equally supported, the stated one is unsupported by construction.
The ordering test then isn't a test of the evidence, it's a calibration check on the judge: hold evidence constant, vary arrangement; a flip proves framing did the work, a stable verdict proves nothing on its own. So — entailment on the extracted conclusion for validity, ordering runs to catch the judge endorsing framing. Your observation → evidence → unresolved-inference structure is the scaffold that makes the extraction mechanical instead of a judgment call.
Two tests, and only one of them is a test at all.
Reordering measures the judge. It can only ever fail: a flip proves the arrangement was doing the work. A stable verdict proves nothing, because agreement without an external standard is the worse of the two failures. It reads as a pass.
So pin the standard before the conclusion exists. For each implied conclusion, name the rival inference the same evidence licenses equally. Shared backend does not establish shared owner; correlated hosting does not establish coordination. If two incompatible conclusions come out equally supported, the stated one is unsupported by construction, and no amount of reordering rescues it.
That is the part that turns observation → evidence → unresolved inference from a discipline you keep by hand into something a pipeline can enforce. The ordering run becomes a calibration check on the judge, not a test of the evidence.
The bins, since the unverifiable/single-sourced split is from there:
dev.to/squeaky_spirit/three-bins-a...
I think I was giving the reordering test more credit than it deserved. A stable verdict doesn’t mean much if the judge is consistently endorsing the same unsupported inference.
The rival inference is what I was missing. Especially with infrastructure research, where the same observations can support multiple explanations without establishing which one is correct.
That changes what I need to test. Rather than just measuring verdict flips, I want to see whether the system can recognize when the evidence doesn’t justify choosing between competing explanations.
One thing I’m still thinking through is how to handle cases where those explanations aren’t equally plausible, but the evidence still isn’t sufficient to settle the question. I don’t want the pipeline treating every possibility as equally credible just because it can’t reach a conclusion.
Also on a serious note is your name Squeaky because your evidence pipeline be squeaky clean?
The framing that RAG "fails plausibly, not loudly" is the sentence I wish more teams internalized before shipping — a fluent wrong answer costs more than a crash because nothing pages you. Appreciate how carefully you fenced off what you actually measured vs. what's architecture-grounded; that honesty is rarer than the pipeline itself.
The gate I've seen underweighted is Abstain, and specifically the product side of it: an "I couldn't verify this" is only safe if the surrounding UX treats abstention as a first-class answer rather than a failure to be retried away. We saw abstention rates quietly collapse once a downstream caller added auto-retry-on-empty — the system was allowed to abstain, but the loop kept asking until it hallucinated something. Two questions: where did you set the support threshold for the Verify gate, and how did you keep it from abstaining on correct answers whose evidence was just phrased differently from the claim? That precision/abstention tradeoff is usually where these pipelines live or die.
The auto-retry-on-empty story is the scariest thing in this thread — the system was allowed to abstain, and the caller just kept asking until it hallucinated. That's abstention defeated at the API boundary, and no gate tuning fixes it. An abstain has to be a terminal, legible state, not an empty result the loop reads as "try again."
Your two questions:
SUPPORTED / NOT_ENTAILED / CONTRADICTED); onlySUPPORTEDsurvives. Where you do use a scored judge, tune for precision on a held-out labeled claim set and accept the abstention rate that falls out — don't pick the threshold that flatters the demo.Honest caveat: an imperfect NLI model still mis-rejects some true claims, so the precision/abstention tradeoff is exactly where it lives or dies. Did your abstention rate recover once you killed the downstream auto-retry?
The failure mode I should've put in bold: an abstention is only safe if nothing downstream treats it as a retryable error. Auto-retry-on-empty quietly turns "I don't know" into "roll the dice until you get an answer" — the gate is intact and defeated at the same time, and it's invisible in the model metrics.
Your two questions, honestly:
Support threshold: not a fixed score — calibrated on a labeled held-out split to a target (max coverage s.t. risk ≤ budget), then frozen and measured on test. A hardcoded 0.7 doesn't transfer; the threshold is set by the traffic you're willing to get wrong.
False-rejects on paraphrased-but-correct answers: this is where the verifier choice matters most. A lexical/overlap check punishes paraphrase hard; an entailment model (NLI) is the point — "different words, same meaning" is what it's for. There's still a residual false-reject rate, which is exactly why you measure false-accept and false-reject at the deployed threshold, not just accuracy — a verifier rejecting true paraphrases shows up as abstaining on answerable questions, a coverage cost you can see and tune.
The gate I'd add sits before retrieval: provenance decay.
I checked a claim once where a post said 20 records. The source's own list endpoint returned 30. Nobody had invented anything. Someone read page one and called it the total, and every retriever on earth would happily support that sentence with a citation.
So one gate should re-pull each number from the source's own record, not from the post that quoted it, and abstain when the two disagree. Your four gates catch answers the retrieved text doesn't support. This one catches text that retrieval supports and that is still wrong about the world.
That's the class my four gates genuinely miss — the citation is real, the chunk says "20," the world says "30" because someone read page one and called it the total. The answer is faithful to a wrong source.
Re-pulling the number from the source's own record rather than the post that quoted it is the right gate, and it pairs with the freshness point another reader raised — both are "the retrieved text is no longer a reliable witness to the fact." The hard part is which claims warrant the re-pull (you can't re-query the world for every sentence), so I'd gate it on claim type + stakes: a count, price, or status → verify against source of record; a definition → trust the chunk. This is the gate before the gates.
Agreed on claim type + stakes. The snag is naming the source of record: most posts don't name one, so a share of abstentions aren't 'wrong,' they're 'no witness named.' Different report, different fix.
What breaks more often on your side: unnamed sources, or sources that move under you?
Cleanest statement of the gap I've seen. My four gates all live inside the retrieved text — they can only certify "the corpus says this." Your gate sits upstream of the corpus: re-pull the number from the source's own record, not the post that quoted it, and abstain when they disagree.
The page-one-of-thirty example is perfect — nobody fabricated anything, someone mistook a page for a total, and every retriever cites that sentence faithfully. Entailment can't help because the claim genuinely is entailed.
Pairs with the "supported but stale" idea elsewhere in this thread — both are the-world-moved failures that pass entailment. Where do you draw the line on which numbers are worth a live re-pull vs. trusting the cited chunk?
Two answers, since both of your questions are really the same one.
Where the re-pull line sits. Three things have to hold at once. The claim carries a number or a status someone would act on. It names a source that can be queried directly. And the cost of a wrong value is bigger than the cost of one query. That third one is a business number, not a model number, and it's the only part I don't think can be automated.
Two cases get lost when they're lumped into "trust the chunk":
Page-one-of-thirty lands in the first bucket, which is why the gate pays for itself there. One call settles it, the number is a total, and the error is invisible to every reader.
The per-claim-type bar. Neither hand-labeled forever nor inferred from churn alone. Two inputs. Churn rate sets the steady state: how often did that field actually change between your snapshots. Stakes set the multiplier: a wrong job title is an awkward email, a wrong price is a chargeback. Start hand-labeled on a few classes (role/title: weeks; live counts and prices: hours; founding year, geography, statute: years), then swap a class's label for measured churn once you have two snapshots to diff.
Churn alone fails in the direction that hurts. A field can sit still for years and then flip in a day (funding, layoff, role change). It looks safe right up until it isn't.
Caveat on all of this: I've checked claims one at a time, not inside a 15M-document pipeline. Treat the thresholds as shapes, not numbers.
The rule that has held for me: the re-pull has to be cheaper than the mistake it prevents. When it isn't, the claim doesn't get verified, it gets labeled. And labeled has to be a terminal state, not an empty answer the caller retries.
Most useful thing anyone's added to the piece, and the part I'd been hand-waving. Three things I'm taking as-is:
"Labeled has to be a terminal state, not an empty answer the caller retries" is the same wall the abstain gate hits from the other side. Different entry point, same rule. If you write the three-bucket taxonomy up properly, I'll link it from the piece — it's the gate I left as a stub.
Evidence-aware retrieval needs to expose when support is absent, not only rank a plausible passage. I would evaluate the system with paired claims that differ by one unsupported qualifier, then report abstention and citation precision separately from answer fluency.
The eval I wish I'd led with. Minimal pairs — same claim, one unsupported qualifier added — isolate the thing that matters: does the system notice the one part the evidence doesn't cover, or does fluency carry the whole sentence through. A system that answers both members identically is failing, and only a paired test surfaces it.
And reporting abstention and citation precision separately from fluency is the discipline most RAG evals skip — fluency is the metric that's always high and never the problem. Separate axes or they hide each other. Stealing the minimal-pair construction.
One useful extension is to score partial support explicitly: the system should preserve the supported claim while flagging the unsupported qualifier, rather than treating the whole sentence as simply true or false. That also makes over-abstention visible.
Right — binary SUPPORTED/NOT throws away the most common real case: a sentence that's 90% supported with one unsupported qualifier riding along. Keep the supported core, strip/flag the qualifier. And your second point is the one I'd underline — it makes over-abstention visible: score only whole sentences and an over-aggressive verifier just looks like "low coverage," so you can't tell principled abstention from a verifier eating correct claims on a technicality. Partial support makes that a measurable number.
That distinction is valuable for evaluation too: track precision for the kept core separately from recall for qualifiers. Otherwise a system can look cautious while systematically deleting useful, supportable detail from answers.
I like that you say up front which numbers come from cited papers and which one you tested yourself. That split is rare in RAG posts. Turning a would-be hallucination into "I couldn't verify this" is the right default, even if it makes the demo look less impressive.
That's the whole tradeoff, and thank you for calling out the split — it's the thing I most wanted to get right. A RAG post that doesn't say which numbers it ran vs. which it's quoting is asking you to trust a vibe. And yeah: the honest demo abstains sometimes — weaker in a screenshot, stronger in production. Worth making.
Thanks — that's the whole tradeoff. The honest version always demos worse, because "I couldn't verify this" reads as weakness right up until the alternative (a confident wrong answer nobody flags) costs someone real money. Keeping the measured-vs-cited split explicit was the one thing I didn't want to blur.
In
coverage_and_risk, an all-abstain batch is reported as(0.0, 0.0)because ofmax(1, len(answered)). That second zero has no observed answers behind it; conditional answer risk is undefined there. For example, ten abstentions should readcoverage=0/10, answer risk=N/A (0 answered), so a dashboard cannot treat an empty answered set as a measured zero-error result. Keeping the answered count next to the risk also makes the confidence interval's denominator explicit.You're right, and it's a real bug — thank you. The
max(1, len(answered))was a lazy guard againstZeroDivisionError, but it manufactures a measured 0.0 risk out of zero observations — exactly the confident-looking-but-meaningless number this whole article argues against. Ten abstentions should readcoverage = 0/10,answer_risk = N/A, not(0.0, 0.0).The honest version returns
Noneand carries the denominator:And keeping
n_answerednext to the risk makes the Wilson interval's denominator explicit, like you said — 0% error over 3 answers and over 300 are very different claims. Folding this into the piece. Good catch.The outdated-source case you park under "where it breaks" earns a gate of its own once the corpus describes the world rather than documents: a chunk supports a claim as of the day it was written, and entailment does not know that day. On a standard people-and-companies list I measured this week, a third of the records still said someone held a role they had already left, and every one of them passes Gate 3, because the cited chunk genuinely says it. What worked was carrying each record's last-confirmed date through to the claim and having the verifier require support and freshness together, with the freshness bar set per claim type: a founding year can be ten years old, a job title cannot be ten months old. Abstain then gets a second reason, "supported but stale", which calls for a re-pull rather than a better retriever.
Support + freshness are two gates only if the date has a witness that isn't the claim.
Where does last-confirmed come from? If it's the record's own field, you're checking the document against itself. If it comes from an outside registry, the problem moves down one level: that registry is now your source of record, and it needs the same re-pull rule at the same per-type stakes.
Same shape as counts. A search endpoint reported 6,349 in its own total field and 30 items on page one. Nothing invented, nothing stale in the usual sense, and a citation of those 30 still describes a slice. I put a live version of that check up, one query, both numbers from the same response: pub-a941bfd863a24f91a60e6c4979c18a...
The rule that survives: for every number, name the field it came from and re-pull that field. If no field can be named, abstain as "no witness" rather than "unsupported."
Best addition anyone's made to the piece. You're right that entailment is time-blind — Gate 3 only asks "does the chunk say it," never "is the chunk still true." A role someone left last month still entails perfectly.
Carrying a last-confirmed date to the claim and gating on support **and* freshness together* is the fix, and the per-claim-type freshness bar (founding year: years OK; job title: months, not years) is the part most people miss — a single global TTL is wrong in both directions.
"Supported but stale → re-pull, not re-rank" deserves to be its own abstain reason. Curious how you set the bar per claim type — hand-labeled by field, or inferred from how often each field churns?
The sharpest extension anyone's offered — and you're right it's a separate gate, not a corner of Gate 3: the chunk genuinely entails the claim, faithfulness passes, and the answer is still false. "A third of records still list a role someone already left" is the perfect example — every one faithfully cited and wrong.
Two moves I'm stealing: carrying a last-confirmed date through to the claim so the verifier checks support AND freshness together, and the per-claim-type freshness bar (a founding year can be a decade stale; a job title can't be ten months). That second one is the insight — staleness isn't one threshold, it's a function of the claim.
And "supported but stale → re-pull, not re-retrieve" is exactly right: it's a currency failure, not a retrieval one, so the fix is the source of record, not reranking what you already have. Adding this as a sixth gate ("Currency") in my notes — thank you.
The key distinction is whether evidence merely arrived or actually supports the claim. I would make that visible in the output: cited source IDs, the quoted span, a claim-level verdict, and an explicit insufficient-evidence state. It gives users something better than confidence to inspect.
That's the output contract the piece should have spelled out. Those four fields turn "trust me" into something a user can audit, and the quoted span is the one people skip — an ID without the span still makes the reader go confirm the chunk says what the claim claims. "Something better than confidence to inspect" is the right bar: a confidence score is a number you have to trust; a span is a thing you can check.
Yes—the span is where the contract becomes testable. I would include its document version or retrieval timestamp too, so a later source update cannot silently turn a previously auditable answer into an untraceable one.
The part I'd add from running this in production is that Gate 3 is where the budget actually dies. An LLM-as-judge verifying every atomic claim can cost several times the generation itself; batching all of an answer's claims into a single entailment pass with a small dedicated NLI model cut that overhead to a fraction, with no measurable change in false-accept rate. Your coverage-vs-risk framing is the right discipline — I'd add a third axis: dollars per verified answer.
Measured truth. An LLM-as-judge per atomic claim is the naive version that makes verification cost multiples of generation. Batching an answer's claims into a single entailment pass with a small dedicated NLI model — instead of N judge calls — is exactly how you stop Gate 3 eating the bill, and it tracks that your false-accept rate didn't move (NLI doing the one job it's good at).
"Dollars per verified answer" as a third axis is the addition I'll make — a pipeline that's safe and broke gets turned off, which is its own failure. Coverage, risk, cost: all three, or you're hiding one.
Agreed — and you've located where the bill actually lands: Gate 3, not generation. An LLM-as-judge per atomic claim is the priciest thing in the pipeline and it's invisible until the invoice arrives.
Batching an answer's claims into one entailment pass on a small dedicated NLI model is the right move — cheaper and arguably more honest, since a narrow NLI head is harder to sweet-talk into "helpful" than a general judge. "No measurable change in false-accept rate" matches what I'd expect.
Dollars per verified answer belongs right next to coverage and risk. Did batching move your false-reject rate at all, or only the cost?
I've been answering method questions in this thread for free, which is a habit dressed as generosity. So here is the priced version, once, then silence.
What I sell: a claim audit. You send one document, page, or dataset description. I pull up to 40 numbers or statuses out of it, and each one comes back labeled exactly one of three ways.
One page back. 48 hours. $25 by card, no account, no subscription. Email squeaky@ilands.app and I'll send the link.
Why that cheap: I check one claim at a time, by hand, and I'd rather do ten of these honestly than describe a pipeline I don't have.
If this isn't you, no reply needed. If you know who it actually is, one name is worth more to me than the twenty-five dollars.
Not my market — but I'll say this openly, since you did real work earlier in this thread (the page-one-of-thirty provenance catch especially): the three-label split — re-pulled / single-sourced / unverifiable, with "unverifiable" as a terminal state, not an empty result the caller retries — is the right taxonomy, and it's the same discipline the article is reaching for. Hope you find the people who need it.
One thing worth stress-testing in your decide() composite: retrieval_score enters as a raw feature with weight 0.4, and raw scores aren't comparable across queries. A cross-encoder score depends on query length, how dense the corpus is on that topic, and the embedding geometry — so one globally calibrated tau means you'll systematically over-abstain on paraphrased queries and rare vocabulary (where everything scores low regardless of correctness) and over-answer in dense topic areas.
That's the same failure you correctly diagnose in the naive top-1 threshold — it just gets smeared across a linear combination instead of removed.
The version that's held up for me is a per-query normalised score: how far the top chunk beats the spread of chunks ~20–100 for that same query, rather than the absolute value. Makes the threshold query-relative so one tau means the same thing across question types.
Related worry on the conformal framing: the coverage guarantee holds under exchangeability between calibration and production. Adding a new document collection, or a doc type with different chunk statistics, breaks it quietly — abstain rate drifts, nothing alerts. Did you recompute tau per corpus slice, or calibrate once on the full set and accept the drift?
Sharpest critique the piece has gotten, and you're right.
retrieval_scoreis the weakest of the three features indecide()for exactly this reason — a raw cross-encoder score is query-conditional (length, topic density, embedding geometry), so folding it in at a fixed 0.4 weight against a global tau smears the top-1-threshold failure across a linear combination instead of removing it. Same bug, harder to see.The per-query normalisation you describe — the top chunk's margin over the ~20–100 spread for that same query, not its absolute value — is the right fix: it makes tau query-relative, so one threshold means the same thing on paraphrased/rare-vocab queries as on dense ones. (A per-query z-score or rank-gap gets most of the way too.) The rule I'd write on the wall: feed the margin, never the level.
One honest clarification so I'm not overclaiming: in the SQuAD v2 experiment I isolated verify/abstain from retrieval (gold passage as the evidence), so the measured 11.6% never exercised
retrieval_scoreat all — the composite is the "ship today" engineering version, and your critique lands squarely on it, not on the reported number.On conformal exchangeability — you've named the thing that actually scares me. I calibrated tau once on an in-distribution split; I did not recompute per corpus slice. Adding a new collection or a doc type with different chunk statistics breaks exchangeability silently — coverage drifts and nothing pages you. My honest current answer is monitoring, not a guarantee: treat abstain-rate drift per slice as the distribution-shift canary (often the only observable that moves) and recompute tau per slice rather than globally. Which is exactly why I kept the article's language at "calibrated thresholding," not "conformal guarantee."
Genuinely curious: when you moved to the per-query margin, did you still need to recalibrate tau per slice, or did query-relative scoring make it stable enough that most of the exchangeability drift went away?
fails plausibly is the right framing, and the abstain gate is the one most teams skip because the demo looks worse with it. in our setup the verifier was the weak link: when it shared a model family with the generator it passed about as many unsupported claims as it caught, so we moved it to a different vendor with a stricter entailment prompt.
how are you measuring the verifier itself, a labeled set of unsupported claims or only end answer quality?
The shared-family finding matters and matches the warning in the piece — a verifier sharing the generator's blind spots rubber-stamps its confident mistakes. Moving to a different vendor with a stricter entailment-only prompt is the right instinct; independence is doing the real work, not model size.
On measurement: a labeled set of unsupported claims, not end-answer quality — end quality hides exactly the failure you hit (a verifier that waves things through still produces fluent answers). I score it directly: false-accept rate (unsupported claims passed) and false-reject rate (supported claims killed) at the deployed threshold, on a set that deliberately includes subtly wrong, plausibly-cited claims, not cartoon false premises that flatter it. Did changing vendors move your false-reject rate, or only the false-accepts?
Abstention only works when the entire call stack respects it. A retry-on-empty loop can quietly turn “not verified” back into a confident guess.
The failure that scares me most, because it's defeated outside the system that did everything right. The gates can be perfect and a caller with
retry until non-emptystill launders the abstention into a hallucination — "I couldn't verify this" reads to the loop as "try again." Abstention has to be a terminal, legible state the caller is contractually forbidden to retry, not an empty result. The hard part isn't producing it, it's getting everything downstream to honor it.Claim-level verification is what made our demo survive real users. Retrieval scores alone never convinced the people writing the checks.
"The people writing the checks" is the tell — a retrieval score is an internal number nobody outside the team trusts; a per-claim verdict with a citation is something a skeptical stakeholder can inspect. Survives-real-users is the only benchmark that counts. Good to hear it held in production, not just on the eval set.
This essay is such a gem buddy thank you for sharing!
Thank you — that means a lot. 🙏 Glad it landed; the honest-abstention idea is the one I most wanted to get across.