"I'm 93% confident" is a string, not a probability

The gap between the demo and the pipeline

In the demo, you ask the model to return a confidence score alongside its answer. It says 0.93. You write if confidence > 0.9: auto_approve() and move on.

In production, that number is just more generated text. Nobody measured it against outcomes. It looks like a probability, it's formatted like a probability, and your code treats it like a probability. That is how false precision becomes automated risk.

A confidence score earns the word "confidence" only when it is calibrated: of all the cases where the system said ~90%, roughly 90% were actually correct. Until you've checked that on your own data, the number is decoration.

Failure mode 1: the threshold that was never tested

Illustrative scenario: a team routes invoice classification through an LLM and auto-processes anything above 0.9. On their eval set, accuracy is 94%. Looks great.

What they never measured is accuracy within the >0.9 bucket. When someone finally slices it, the high-confidence bucket is barely better than the low-confidence one. The model emits 0.9+ on almost everything. The threshold wasn't filtering anything; it was a rubber stamp with a decimal point.

Failure mode 2: one accuracy number hiding two systems

If your system can defer (hand off to a human, ask for clarification, fall back to a rule), then you're not evaluating one number. You're evaluating a trade-off:

• Coverage: how often does the system answer instead of deferring?
• Selective accuracy: when it answers, how often is it right?

Push the threshold up and coverage drops while selective accuracy (hopefully) rises. Whether it actually rises is the whole question. Here's the minimal version of measuring it:

import numpy as np

def coverage_accuracy_curve(conf, correct, thresholds):
    conf, correct = np.asarray(conf), np.asarray(correct)
    rows = []
    for t in thresholds:
        answered = conf >= t
        coverage = answered.mean()
        sel_acc = correct[answered].mean() if answered.any() else float("nan")
        rows.append((t, round(coverage, 3), round(sel_acc, 3)))
    return rows

def calibration_bins(conf, correct, n_bins=10):
    conf, correct = np.asarray(conf), np.asarray(correct)
    edges = np.linspace(0, 1, n_bins + 1)
    for lo, hi in zip(edges[:-1], edges[1:]):
        m = (conf >= lo) & (conf < hi)
        if m.any():
            print(f"[{lo:.1f}, {hi:.1f})  n={m.sum():4d}  "
                  f"stated={conf[m].mean():.2f}  actual={correct[m].mean():.2f}")

If "stated" and "actual" drift apart in the bins you care about, your threshold is lying to you.

Failure mode 3: the global threshold

Even a calibrated score can be calibrated on average and miscalibrated everywhere that matters. Illustrative scenario: overall, the 0.9 bucket is 91% correct. Split by language and the English tickets sit at 95% while the Greek and German ones sit far lower. Split by input quality and the clean PDFs are fine while the scanned, OCR'd ones fall apart.

One threshold across all of that averages away exactly the failures you built the deferral path to catch. Run the same curve per slice:

for slice_name, idx in slices.items():   # e.g. by task type, language, source quality
    print(slice_name, coverage_accuracy_curve(conf[idx], correct[idx], [0.7, 0.8, 0.9]))

If the curves differ meaningfully, you need per-slice thresholds, or you need to stop trusting the score for that slice at all.

Failure mode 4: verbalized vs. measured confidence

Asking the model "how sure are you?" and reading token log-probs are different signals, and neither is calibrated by default. Self-reported numbers in particular tend to cluster on round, reassuring values. Treat any source of confidence (verbal, log-prob, ensemble agreement, a separate verifier) as a hypothesis to test, not a fact to consume.

The reframe

Confidence isn't a model property. It's a system property you have to measure. The real question isn't "does the model feel sure?" It's "is the system's decision to proceed reliable for this task, in this context, on this kind of input?"

Practical checklist

1. Log every decision with its confidence, the slice metadata, and the eventual ground truth.
2. Plot coverage vs. selective accuracy, globally and per slice.
3. Check calibration bins in the range where your threshold actually sits.
4. Re-run it when the model, prompt, or input distribution changes. Calibration doesn't survive a model swap.

How are you deciding when your LLM pipeline is allowed to act without a human? Is anyone here actually running per-slice thresholds in production, or is everyone on a single global cutoff?

Comments

[Sina Rezaei]

The payment reconciliation case is where this gets interesting. Imagine an LLM extracting beneficiary details from supplier invoices and matching them against ERP records. It reports 93% confidence in a match, so the pipeline automatically approves it. But if that confidence comes from the model's own response rather than a validated probability, you've basically automated a financial decision using a number that hasn't earned any trust.

I'd go a step further than checking calibration across the whole dataset. Separate clean digital invoices from OCR-heavy scans, new suppliers from established ones, and exact matches from cases involving similar company names or changed bank details. Then measure the actual error rate in each group and route uncertain or high-risk matches for review.

One thing worth watching is sample size, too. A bucket with 93% observed accuracy based on 30 cases doesn't give you the same evidence as one based on thousands. And even a well-calibrated score won't protect you from a new input distribution.

The real test isn't whether the model can produce a convincing confidence score. It's whether that score can reliably tell your system when it's safe to act and when it should stop.