A Tiny Probability Puzzle Every Developer Should Try

Here's a puzzle I like to give developers, because it's short, slightly uncomfortable, and teaches something you'll use for the rest of your career.

The puzzle: Over the next month, write down 20 predictions about your work. Each one gets a probability. "70% this PR merges by Friday." "90% the deploy goes out without a rollback." "40% the flaky test fails again today." At the end of the month, check which happened.

Now the question: when you said 90%, did 90% of those things happen?

If you've never done this, I'll make a forecast of my own: you're overconfident. Most people are. Developers are especially prone, because we estimate things constantly (story points, timelines, "this'll take five minutes") and rarely look back at the receipts.

What calibration actually means

Being calibrated means your confidence matches your hit rate. If you're perfectly calibrated, then across everything you labelled "70%," about 70% came true.

Notice that calibration is not the same as accuracy. Someone who says "50%" about everything is perfectly calibrated on a coin flip and completely useless. What we want is to be calibrated and decisive: confident when we should be, uncertain when we should be.

Let's simulate it

Before you trust your own data, it helps to see what calibration looks like in a controlled world. The script below simulates two forecasters:

• Honest Hannah, whose stated probability is the true probability.
• Overconfident Omar, who sees the same truth but pushes his estimate toward the extremes.

Then we bucket the forecasts and compare stated confidence to observed frequency, and compute a Brier score (lower is better).

import random

random.seed(42)

def overconfident(p, strength=1.8):
    """Push a probability away from 0.5 toward 0 or 1."""
    odds = (p / (1 - p)) ** strength
    return odds / (1 + odds)

def simulate(n=10_000):
    hannah, omar = [], []
    for _ in range(n):
        true_p = random.uniform(0.05, 0.95)
        outcome = 1 if random.random() < true_p else 0
        hannah.append((true_p, outcome))
        omar.append((overconfident(true_p), outcome))
    return hannah, omar

def brier(forecasts):
    return sum((p - o) ** 2 for p, o in forecasts) / len(forecasts)

def calibration_table(forecasts, bins=10):
    buckets = [[] for _ in range(bins)]
    for p, o in forecasts:
        idx = min(int(p * bins), bins - 1)
        buckets[idx].append((p, o))
    rows = []
    for b in buckets:
        if b:
            stated = sum(p for p, _ in b) / len(b)
            observed = sum(o for _, o in b) / len(b)
            rows.append((stated, observed, len(b)))
    return rows

hannah, omar = simulate()

for name, data in [("Hannah", hannah), ("Omar", omar)]:
    print(f"\n{name}  Brier score: {brier(data):.4f}")
    print(" stated  observed    n")
    for stated, observed, n in calibration_table(data):
        print(f"  {stated:5.2f}    {observed:5.2f}  {n:5d}")

Run it and look at Omar's table. In his top bucket he says something like 0.97, but those events happen closer to 0.9. In his bottom bucket he says 0.03, but they happen more like 0.1. His Brier score is worse than Hannah's, even though he had exactly the same information. The only difference is how he translated belief into numbers.

That's the uncomfortable lesson: you can lose accuracy without knowing less. You just have to be a little too sure.

Try it on yourself

You don't need a fancy setup. A text file works:

2026-11-02 | 0.80 | PR #412 merged by Friday | 1
2026-11-02 | 0.60 | No pager alerts this weekend | 0

After 20 to 30 entries, feed the probabilities and outcomes into calibration_table and brier. With small samples the buckets will be noisy, so use 3 to 5 bins instead of 10.

What people usually discover

1. The 90% bucket is a liar. Things you were "basically certain" about fail surprisingly often.
2. Estimates cluster. Many people only ever use 50%, 80%, and 95%. Forcing yourself to pick 65% or 72% sharpens thinking.
3. Base rates beat vibes. "How often has a deploy like this rolled back before?" is a better starting point than "it feels fine."

Why this matters for developers

Estimation is a huge part of our job, and it's rarely trained. Calibration gives you a feedback loop: not "was I right?" but "was my confidence right?" That's a much kinder and more useful question. It turns estimates from a source of anxiety into a skill you can measure and improve.

So: 20 predictions, one month, one small script. Post your Brier score in the comments if you're brave. I'll go first next month.