The most expensive bug in an LLM system is not the output that is obviously broken. That one you catch. It is the fluent, well formed, confidently wrong answer that reads exactly like a correct one and walks straight into your system of record because nothing was standing between the model and production.
"Reads correct" and "is correct" are two different properties, and the model only optimizes the first. So the answer that is almost right, the near miss, comes back just as polished as the answer that is exactly right. You cannot tell them apart by looking. Neither can the model.
Similarity is not confidence
When teams want to gate on that, they reach for a number. Usually one of two: the model's own reported confidence, or the similarity score of whatever chunk it retrieved. Both feel like confidence. Neither one is.
Ask a model how sure it is and you get more generated text, sampled from the same distribution that just produced the wrong answer. It is a guess about a guess. And a similarity score measures proximity, not truth. The near miss scores high precisely because it sits close to the right answer. That is the whole trap: a similarity cutoff hands its highest marks to the single most dangerous output in the system, the confident near miss, and waves it through.
Compute it from what you can check
The way out is to stop reading confidence off the model and start computing it from signals you can verify without trusting the model at all. Three of them carry most of the weight.
Grounding asks whether the output is actually entailed by the evidence you gave it, not whether it reads well. An answer nobody can trace back to a source is not trustworthy, however clean it looks. A lexical-overlap baseline gets you moving, but the real seam is a (claim, evidence) -> float function, so you can swap in an NLI model or a cross-encoder the moment overlap is too blunt. Entailment over proximity.
Agreement resamples the generator on the same input and measures whether the answer survives. A model that flaps between three answers when you ask three times is telling you something a single clean draw hides. A minority draw is low confidence even when it parses perfectly. This is the one signal that costs you extra generations, so you price it in on purpose.
Validation is the cheap one, and the one people skip. Does the output satisfy the schema you declared, every required field present and the right type? A structurally invalid output should never reach the auto-accept ceiling no matter how good the other signals look, so one missing field drags the score down instead of passing silently.
The gate, as code
I wrote this up as a small reference implementation, confidence-gate, so the pattern has something runnable behind it. The generator goes in as a plain callable, which keeps the whole thing deterministic and runs it with no API key. Wiring it looks like this:
from pydantic import BaseModel
from confidence_gate import (
AgreementSignal, GroundingSignal, Policy,
ReviewQueue, Router, Task, ValidationSignal,
)
class Invoice(BaseModel):
invoice_number: str
vendor: str
total: str
date: str
def generator(prompt: str) -> str:
# Swap in a real model call. The gate does not care where the text
# comes from, only whether it survives the signals.
return '{"invoice_number": "INV-4021", "vendor": "Northwind Supplies", "total": "1840.00", "date": "2026-02-11"}'
queue = ReviewQueue(":memory:")
signals = [ValidationSignal(Invoice), GroundingSignal(), AgreementSignal(generator, k=3)]
router = Router(generator, signals, Policy(), queue)
verdict = router.route(
Task(
task_id="inv-1",
prompt="extract invoice fields",
evidence="Invoice INV-4021 from Northwind Supplies. Date 2026-02-11. Total 1840.00 USD.",
)
)
print(verdict.decision, verdict.aggregate_confidence)
Run the bundled demo over three documents, one clean, one ambiguous, one with no supporting evidence, and it prints the decision for each. Captured output:
TASK DECISION CONF SIGNALS
----------------------------------------------------------------
northwind-clean AUTO_ACCEPT 1.00 validation=1.00 grounding=1.00 agreement=1.00
globex-partial HUMAN_REVIEW 0.50 validation=0.75 grounding=0.50 agreement=0.67
unknown-source ABSTAIN 0.00 validation=1.00 grounding=0.00 agreement=1.00
review queue: 1 pending
#1 globex-partial conf=0.50 :: min aggregate 0.500 (accept>=0.75, abstain<=0.35) -> human_review
The middle row is the whole argument. globex-partial validates against the schema except for one missing field, only half of it is supported by the evidence, and the generator does not fully agree with itself on resampling. No single number screams "wrong", so the gate refuses to auto-accept and refuses to abstain. It routes the item to a person and keeps the receipt.
The bottom row is the one that would have burned you. unknown-source is structurally perfect, validation 1.00, and the model reproduces it on every resample, agreement 1.00. By any signal you could read off the model, it is maximally confident. Grounding is 0.00 because nothing in the evidence supports it. A confident, self-consistent answer with no external support, and the gate abstains.
Tune on the cost of a false accept
Look at how the aggregate falls out. It is the minimum of the signals, not the average. That is deliberate, and it comes from the cost structure rather than a taste for being strict. A cache miss or a re-run costs seconds. A false accept ships a confident wrong answer into the record with no model left in the loop to hedge it. Those two outcomes are not symmetric, so you do not tune the threshold to hit some acceptance rate. You tune it on what a single false accept costs you. A gate is only as strong as its weakest verified signal, so one failing signal is enough to deny auto-accept. Abstaining is cheap. Being confidently wrong is the bill you do not see coming.
That leaves three outcomes instead of two. Auto-accept when the external signals agree. Human review, with the full audit trail, when they are mixed and a person should look. Abstain when there is no support at all, because refusing to answer beats guessing when you cannot check the guess. The model proposes, the code disposes.
What this does not do
It does not make a wrong answer right. It refuses to ship one silently, which is a smaller and far more achievable goal. If your evidence is itself wrong, grounding will happily confirm a wrong answer against it, so the quality of your source text just became part of your trust boundary. The default entailment is lexical overlap, which is blunt: it catches "the answer cites nothing in the source" but not "the answer inverts the one number that mattered", so anything real gets a stronger function through that seam. Agreement costs you K generations per item, which is real latency and real money traded for a convergence signal. And none of this is a product with uptime promises. It is the shape of the design, made runnable.
A confident wrong answer and a confident right one are the same string until something outside the model checks. Be that something.
Top comments (4)
Strong direction. My concern is that combining several imperfect signals can still produce a score people trust too much. Agreement, grounding, and validation are useful checks, not proof. The approach would be stronger with thresholds calibrated on labelled data and evaluated by false-accept risk versus automation coverage.
Fair concern, and it is the exact failure mode the design guards against, with two moves.
First, the aggregator is a minimum, not a mean, precisely because a mean lets a strong signal paper over a weak one and manufacture the over-trust you describe. Under min the score can never exceed the weakest verified signal, so three checks that are "not proof" cannot combine into a proof-shaped number. It is a conservative floor, not a probability of correctness, and should never be read as one: 0.6 means "cleared the checks to this degree," not "60 percent likely right."
Second, your calibration point is the intended evaluation, not an add-on. The threshold is set on labeled data and reported as false-accept rate on the auto-accepted band against the coverage you bought at that cut. The score only has to rank items well enough that the cut sits where false-accepts stay inside tolerance. Calibrate on the cost of a false accept, not acceptance rate, and it stops being a number to trust and becomes a routing coordinate. Checks plus a min plus a cost-calibrated cut, not proof: that is the whole claim.
That distinction makes sense, especially treating the score as a routing coordinate rather than a probability.
I think the remaining risk is upstream of the aggregator:
minprevents compensation between signals, but it cannot protect against correlated blind spots, jointly miscalibrated checks, or failures outside the checks’ coverage. In those cases, every signal may clear its threshold while the answer is still wrong.So the key evidence would be whether the false-accept bound holds across domains, model changes, and distribution shift, not only on the calibration set. I’d be interested in how you detect when the routing policy itself needs recalibration.
You are right, and it is the honest limit: min defends against compensation, not coverage. If the checks share a blind spot or the failure sits outside all of them, min just takes a clean floor over the wrong set. The score is a floor conditional on the checks covering what matters and not failing correlated with the answer. Only an independent check or a ground-truth signal fixes that, and correlated blind spots are the residual I would not claim to have solved.
So the bound does not transfer for free: it is measured on the calibration set, and a model swap breaks it by construction. The rule for detecting drift is that the detector has to be external to the checks, or you rebuild the self-report problem one level up. Concretely: label a random sample of the AUTO_ACCEPT band in production and watch the realized false-accept rate against tolerance, that is the only direct detector, and treat any model change as an automatic recalibration event, unknown until re-measured.