Your Agent's Confidence Score Is Not a Probability

Ask an agent how sure it is and it will happily tell you. "Confidence: 0.92." It looks like a probability. It renders nicely in a dashboard. Teams wire it into routing logic: high confidence, auto-approve; low confidence, send to a human. It feels rigorous.

It is not rigorous. A self-reported confidence score is the agent grading its own homework, and it is one of the most seductive false signals in production agentic systems. If you are gating anything on it, you are trusting the defendant's opinion of their own alibi.

Where the number actually comes from

When an LLM emits confidence: 0.92, that number is not a calibrated posterior. It is another token sequence, generated by the same forward pass that produced the answer you are unsure about. It shares a substrate with the output it is describing. If the model hallucinated a file path, the same weights that invented the path will cheerfully assign it 0.9 confidence, because from the inside, a confident fabrication and a confident fact are indistinguishable.

This is not a prompt-engineering problem you can fix with "be honest about your uncertainty." You can push the distribution around, but you cannot make a model's self-report into independent evidence, because there is no independent ground truth in the loop. The grader and the graded are the same network.

The independence axis, applied

This is exactly why, when we build evals at agent-eval, we rank evidence on an independence axis — independent to corruptible — not a cost axis of cheap to expensive. Three tiers:

• Tier 1 — externally observable proof the agent can't forge. The JSON parses. The file exists on disk. The code compiled. The tests passed. The call returned within the timeout. The output is non-empty. None of this can be faked by a confident model, because you are checking reality, not asking an opinion.
• Tier 2 — statistical signal against a baseline the agent didn't author. Embedding similarity between the output and the actual task. Length and repetition checks. Did the diff actually change anything. The agent didn't write the baseline, so it can't game the comparison.
• Tier 3 — model-as-judge. A shared-substrate opinion. A signal, never a verdict.

A self-reported confidence score is not even Tier 3. Tier 3 at least lets a separate model inspect an artifact. Self-confidence is the judged model judging itself in the same breath — maximally circular. It belongs at the very corruptible end of the axis.

Two facts that fall out of this

Tier 1+2 are the real-time gate. They are deterministic, roughly free, and fast, so they can sit in the hot path and block a bad run before it ships. Tier 3 is offline-only: metered, slow, non-deterministic. A judge cannot live in your latency budget. And self-reported confidence, despite looking cheap enough to gate on, is corruptible enough that gating on it is worse than gating on nothing — because it gives you false comfort.

A model judging another model's reasoning is circular. Tier 1+2 can run over agent trajectories. Tier 3 cannot — if you let a judge grade the reasoning steps, judge and judged share a substrate and there is no independent ground truth. So Tier 3 may only inspect artifacts the judged agent didn't get to write. Self-confidence violates this rule harder than anything else: it is a claim about the reasoning, authored by the reasoner.

What to do instead

Don't route on the agent's opinion of itself. Route on independent checks. Here is the shape of it in TypeScript:

type AgentOutput = {
  answer: string;
  filePath: string;
  selfConfidence: number; // present, and deliberately ignored for gating
};

type GateResult = { pass: boolean; tier: 1 | 2; reason: string };

async function gate(out: AgentOutput, task: string): Promise<GateResult> {
  // Tier 1: unforgeable proof
  if (!out.answer.trim()) return { pass: false, tier: 1, reason: "empty" };
  if (!(await fileExists(out.filePath)))
    return { pass: false, tier: 1, reason: "hallucinated path" };

  // Tier 2: statistical signal vs a baseline the agent didn't author
  const sim = await cosineSim(embed(out.answer), embed(task));
  if (sim < 0.55)
    return { pass: false, tier: 2, reason: `off-task (sim=${sim.toFixed(2)})` };

  // out.selfConfidence is never consulted. It's the defendant's alibi.
  return { pass: true, tier: 2, reason: "cleared independent checks" };
}

Notice selfConfidence is present in the type and never read in the gate. That is the point. You can log it, correlate it against outcomes offline, even discover it's anti-correlated with correctness — but it does not get a vote in the hot path.

Ship the 80%, then let a judge whisper about the rest

Most production failures are boring and mechanical: stale data, a crash, malformed output, a hallucinated path, an empty result. Every one of those is caught at Tier 1+2 alone, deterministically, for about zero dollars. Reserve the model-as-judge for the roughly 20% subjective tail — tone, helpfulness, whether an explanation is actually clear — and label its output honestly as "opinion, not evidence." A judge that says 7/10 is a suggestion for a human, not a gate.

You can't gate what you can't see

All of this assumes you can actually inspect what the agent did — the real inputs after resolution, the real tool outputs, the real intermediate steps. That is the other half of the workflow. AgentLens captures the trace: every model and tool step, resolved inputs, raw outputs. agent-eval scores and gates the output; AgentLens gives you the unforgeable, agent-didn't-author trace data for Tier 1+2 to score against — and the debugging surface for when a gate goes red and you need to know why.

The pairing matters here specifically because the whole failure mode of this post is trusting the agent's narration of itself. AgentLens replaces the narration with the record. agent-eval judges the record, not the narration.

Self-reported confidence is the narration. Stop routing on it. Route on proof.