Does the Model Know When It's Wrong? Building an LLM Confidence Calibration Tool

You can build this yourself — it's ~400 lines of Python and one Streamlit
dashboard. Here's the full technical walkthrough.

The question that matters

Every LLM answers with the same self-assured tone whether it's reciting a
well-known fact or confidently hallucinating a plausible-sounding one. That's the core problem of calibration: does the probability a model reports for an answer actually match the probability that the answer is correct?

If a model says "I'm 90% sure" and it's right 90% of the time, it's perfectly calibrated. Most aren't. Most are overconfident — they say 90% when they're really right 60% of the time. This isn't just an academic curiosity. In production, an overconfident model that's wrong is worse than a humble model that abstains: one quietly corrupts your output, the other tells you it doesn't know.

So I built a small, self-contained tool that answers one question about any OpenAI-compatible model: does it know when it's wrong?

---

What it does

The tool asks a model ~65 general-knowledge multiple-choice questions and, for each one, captures two things:

1. The model's chosen answer (A–D)
2. Its self-reported confidence (0.0–1.0)

It never shows the model the correct answer. Then it buckets every answer by reported confidence, computes the actual accuracy inside each bucket, and plots accuracy against confidence. The distance from the y = x diagonal is a direct, visual measure of miscalibration.

There are two entry points:

• calibration.py — a dependency-light CLI (just openai, matplotlib, python-dotenv).
• app.py — a Streamlit dashboard that swaps the raw openai client for the OpenAI Agents SDK.

---

Architecture

The key design decision: the dataset and the analysis engine are shared.
calibration.py owns the pure functions — analyze(), make_figure(),
make_takeaway() — and app.py imports them. The only thing that differs
between the two entry points is how the model is called. That keeps the
science identical and lets the CLI and dashboard never drift apart.

---

The dataset

Each question is a plain dict with four unique options and one correct letter.
I deliberately balanced the answer distribution (16 A's, 16 B's, 17 C's, 16 D's)
so a model can't exploit positional bias and inflate its accuracy:

{"q": "What is the capital of Australia?",
 "options": ["Canberra", "Sydney", "Melbourne", "Perth"], "answer": "A"}

65 questions spread across trivia, math, science, and history — enough for
meaningful per-bucket stats without costing a fortune in API calls.

---

Asking for an answer and confidence

The key trick is that confidence must be elicited explicitly, not inferred.
The system prompt forces a strict JSON shape so both fields parse cleanly:

SYSTEM_PROMPT = (
    "You are a helpful assistant answering general-knowledge multiple-choice "
    "questions. Always respond with ONLY a single JSON object of the exact form "
    '{"answer": "A", "confidence": 0.85}, where "answer" is the letter of the '
    "correct option (A, B, C, or D) and \"confidence\" is a number between 0 and 1 "
    "representing how confident you are that your chosen answer is correct. "
    "Do not include any other text, markdown, or explanation."
)

Parsing is defensive — it handles JSON, markdown-fenced JSON, and free-text
fallbacks, and any unparseable response is skipped and counted rather than
crashing the run:

def parse_answer_letter(text):
    m = re.search(r'"answer"\s*:\s*"?([A-Da-d])"?', text)
    if m:
        return m.group(1).upper()
    m = re.search(r"\b([A-Da-d])\b", text)
    return m.group(1).upper() if m else None

---

The analysis engine

Everything downstream is pure math over a list of (confidence, correct)
tuples. analyze() produces the per-bucket data and the abstention sweep:

def analyze(results):
    n_total = len(results)
    n_correct = sum(1 for _, c in results if c)
    overall_acc = n_correct / n_total
    overall_conf = sum(c for c, _ in results) / n_total
    # ... bucket into [0.0-0.2, 0.2-0.4, ...], compute mean conf + accuracy ...
    # ... sweep thresholds t in [0.0, 0.3, 0.5, 0.7, 0.9] for abstention ...
    return {...}

And make_figure() renders the curve — accuracy on the y-axis, mean reported
confidence on the x-axis, with the ideal y = x diagonal as the reference:

fig, ax = plt.subplots(figsize=(7, 7))
ax.plot([0, 1], [0, 1], "--", color="gray", label="Perfect calibration (y=x)")
ax.plot(xs, ys, "o-", color="#1f77b4", label="Model accuracy")

---

The OpenAI Agents SDK path

The dashboard doesn't call the model with the raw openai client. It uses the
OpenAI Agents SDK, which gives us a clean Agent abstraction and structured
output — the answer and confidence come back as a typed Pydantic model
instead of a string we have to regex:

from agents import Agent, OpenAIChatCompletionsModel, Runner
from openai import AsyncOpenAI
from pydantic import BaseModel, Field

class AnswerConfidence(BaseModel):
    answer: str = Field(description="The chosen option letter: A, B, C, or D")
    confidence: float = Field(ge=0.0, le=1.0, description="Confidence 0-1")

client = AsyncOpenAI(base_url=BASE_URL, api_key=API_KEY)
model = OpenAIChatCompletionsModel(model=MODEL, openai_client=client)

agent = Agent(name="quizzer", model=model,
              instructions=AGENT_INSTRUCTIONS,
              output_type=AnswerConfidence)

Because OpenAIChatCompletionsModel takes an AsyncOpenAI client with a custom
base_url, the same code works against OpenAI, a proxy, or a local
OpenAI-compatible endpoint.

Structured output with a safety net

Structured output serializes output_type into a JSON-schema
response_format. Not every OpenAI-compatible endpoint supports that, so the
app tries structured output first and falls back to a plain-text agent that
parses the JSON manually — broad compatibility without sacrificing the clean
path:

try:
    result = _run_async(Runner.run(_make_agent(AnswerConfidence), prompt))
    if isinstance(result.final_output, AnswerConfidence):
        return result.final_output.answer.upper(), result.final_output.confidence
except Exception:
    pass
# fallback: plain-text agent + manual parse

---

The asyncio gotcha (the part that will save you hours)

Here's the bug I hit that you should know about. The SDK's Runner.run_sync
internally calls asyncio.get_event_loop(). That works fine in a normal script,
but Streamlit runs your script in a non-main thread named
ScriptRunner.scriptThread, where no event loop exists — so get_event_loop()
raises:

RuntimeError: There is no current event loop in thread 'ScriptRunner.scriptThread'

The fix is to create a persistent event loop for that thread and drive the
async Runner.run on it — and to set it as the current loop so any internal
get_event_loop() calls also succeed:

_loop = None

def _get_loop():
    global _loop
    if _loop is None:
        _loop = asyncio.new_event_loop()
        asyncio.set_event_loop(_loop)   # makes get_event_loop() work too
    return _loop

def _run_async(coro):
    return _get_loop().run_until_complete(coro)

# then, instead of run_sync:
result = _run_async(Runner.run(agent, prompt))

This is a genuinely reusable pattern for anyone combining the OpenAI Agents SDK
(or any asyncio-based library) with Streamlit.

---

Reading the results

A typical run prints something like:

Overall accuracy: 0.65   Mean reported confidence: 0.82

CALIBRATION TABLE (by confidence bucket)
Bucket         Mean conf  Accuracy     n
[0.0-0.2]          --        --     0
[0.2-0.4]       0.30       0.33     6
[0.4-0.6]       0.51       0.47    15
[0.6-0.8]       0.71       0.63    21
[0.8-1.0]       0.88       0.76    23

TAKEAWAY: OVERCONFIDENT: mean confidence 0.82 exceeds accuracy 0.65

The tell is right there: the model reports 82% average confidence but is
only 65% accurate. And it's worst exactly where it's most confident — in the
[0.8-1.0] bucket it says 88% but delivers 76%. That's the signature of
overconfidence: the curve bends below the diagonal.

The abstention analysis then shows the practical fix — if you only keep
answers where confidence ≥ 0.9, accuracy climbs (because you drop the
low-confidence wrong answers), at the cost of coverage:

Threshold   Accuracy  Coverage   n
      0.0      0.65   100.0%    65
      0.5      0.72    72.3%    47
      0.7      0.78    49.2%    32
      0.9      0.84    24.6%    16

This is the accuracy/coverage trade-off every production system has to make.

---

Key takeaways for developers

1. Confidence must be elicited, not inferred — ask the model for it explicitly and force a parseable format.
2. Calibration is a measurable property — the distance from the y = x diagonal tells you more about trustworthiness than raw accuracy ever will.
3. Structured output + a plain-text fallback is the sweet spot for OpenAI-compatible endpoints of unknown capabilities.
4. Streamlit + asyncio libraries need a persistent event loop — remember asyncio.new_event_loop() + set_event_loop + run_until_complete.
5. Abstention is a real lever — thresholds convert overconfidence into reliability at the cost of coverage.

---

Future work

• Expected Calibration Error (ECE) — the standard quantitative summary.
• Pluggable datasets — MMLU, TriviaQA, or custom JSON, selectable in the UI.
• Temperature sweeps — how does sampling temperature shift confidence?
• Per-category curves — where is the model most overconfident?
• Parallelized runs with asyncio.gather for speed.

Code & build: https://www.dailybuild.xyz/project/241-calibri