My Free Model's Confidence Score Was a Liar. Here's the Probe.

The model said it was 93% sure. It was wrong. That wasn't a surprise — free models are sloppy with probabilities. But I wanted to know exactly how sloppy, because "confidence" is the one number engineers actually trust when they're in a hurry.

So I ran a calibration test. Not a benchmark, not a head-to-head against GPT-4. Just a tiny, boring experiment that answers one question: when this model says a number, does that number mean anything?

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The setup

I used MonkeyCode's free model access and the free server option. No paid tier, no special endpoint. Just the ordinary free lane.

I wrote 20 short Python and JavaScript snippets. Each one had exactly one bug — a bad index, a missing await, an off-by-one, a false condition. Then I gave them to the model with this prompt:

Does this code have a bug? Respond with JSON only.
{"bug": true or false, "confidence": integer 0-100}

I ran each snippet five times with temperature=0. That gave me 100 responses to analyze. Doing it five times mattered, because free servers are nondeterministic even at zero temperature. Sometimes the same snippet gets two different answers.

The harness itself is simple. This is the core function:

import os
import requests
import json

API_URL = os.getenv("MONKEYCODE_API_URL")
API_KEY = os.getenv("MONKEYCODE_API_KEY")
MODEL = os.getenv("MONKEYCODE_MODEL")

def ask(snippet: str) -> dict:
    resp = requests.post(
        API_URL,
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={
            "model": MODEL,
            "temperature": 0,
            "messages": [
                {"role": "system", "content": "You are a code reviewer."},
                {"role": "user", "content": (
                    "Does this code have a bug? Reply with JSON only. "
                    "{\"bug\": true or false, \"confidence\": integer 0-100}\n\n"
                    + snippet
                )}
            ],
        },
        timeout=30,
    )
    content = resp.json()["choices"][0]["message"]["content"]
    return json.loads(content)

Then I binned the responses by confidence and compared each bin to the actual answer. The math is five lines of pandas.

What the numbers said

Here's my run, aggregated across all 100 responses:

Confidence range	Responses	Accuracy
90–100	42	38%
70–89	36	47%
50–69	22	55%
0–49	10	60%

Read that slowly. When the model was most confident, it was least correct. The 90–100 crowd landed at 38% accuracy — worse than a coin flip. Meanwhile, responses with low confidence were right more often than not. That's not a small calibration error. That's an inverted signal.

The model wasn't stupid. It found obvious bugs with 100% accuracy and high confidence. Syntax errors, undefined variables, resources that were never closed. Those are the easy ones. But the subtle bugs — race conditions, off-by-one on an edge, a wrong comparison operator — those got a confident "yes, this is fine" far too often. The confidence score behaved like a fluent guesser, not a probability.

The reproducible part

You don't have to trust my numbers. Build your own case list and run the same script. The key is to separate easy bugs from hard ones, because free models are genuinely useful on the easy end and dangerously confident on the hard end.

Here's a sample of the kind of cases I used:

cases = [
    # Python: list comprehension shadowing the loop variable
    ("python", "items = [1, 2, 3]\nresult = [i for i in items if i % 2]\nprint(sum(i for i in items))", True),
    # JS: missing await in async function
    ("javascript", "async function getTotal() {\n  const data = fetch('/api/data');\n  return data.json();\n}", True),
    # Python: correct code
    ("python", "def add(a, b):\n    return a + b", False),
]

Each tuple is (language, code, has_bug). Feed that into the ask() function, collect the confidence score, and calculate accuracy per bin. The aggregate will almost certainly show the same L-shaped curve: high confidence, poor calibration.

Why this matters

The danger is not that free models are wrong. Every model is wrong sometimes. The danger is that the confidence number gives your brain permission to skip the code review. A 95% confidence score feels like a green light, so you paste the result into a PR and move on. That's how bugs ship.

What should you do differently? Treat confidence as a "needs human look" flag, not a pass/fail gate. If the model says 80% or higher, read the snippet yourself. If it's unsure, that's often where it actually knows something. The uncertainty is more informative than the assertion.

Limitations

This is one model, one prompt format, and 20 hand-written snippets. It is not a model benchmark. Server load, prompt wording, and even the language of the snippet can move the numbers. The free server in particular felt faster in my tests, but speed comes at a cost: more retries, occasional timeouts, and no guarantee that you're hitting the same model on every call.

Also, my snippets were biased toward logic bugs. If your codebase is mostly integration glue, your calibration curve will look different — possibly worse.

Who shouldn't use this approach

Skip this probe if you need airtight, auditable probabilities. For a medical device or a financial trading system, this kind of free-model confidence is worse than useless; it's a liability. Use a model with documented calibration, or better yet, don't use a model to decide.

But if you're a solo dev shipping a side project, or a team trying to triage a backlog of legacy code, the free tier gives you enough tokens — I've seen the 10 million claim in the docs — to waste on experiments like this. The only thing you lose is an afternoon of guessing.

My script is generic. Point it at any OpenAI-compatible endpoint, set a few environment variables, and you're running your own calibration probe. That's the real takeaway: confidence is a metric, so measure it like one.

If you want to see how wrong your free model is, don't trust my numbers. Run the probe. The code above is all you need.