Uncertainty Quantification in AI (Part - 1)

Karthik Amirapu — Tue, 28 Jul 2026 09:11:09 +0000

The following is series of posts on Uncertainity quantification in AI systems and discussion on some of the statistical methods to overcome

Chapter 1 — Why AI Needs Uncertainty

1.1 The problem in one sentence

Note- AI confidently hallucinating the spelling of 'trustworthy' in the image above - kind of proving the point of this article, it cant be trustworthy (pun intended) all the time.

Every prediction a machine learning system makes is a guess about something it has never seen before. A model that reads a chest X-ray and says "pneumonia" is guessing. A model that predicts tomorrow's electricity demand is guessing. This is not a criticism — guessing from patterns in past data is exactly what these systems are built to do, and they often do it remarkably well. The difficulty is not that they guess. The difficulty is that most of them guess without ever telling us how much to trust the guess.

Imagine a friend who answers every question you ask with total confidence, whether they know the answer or not. When they happen to be right, they are useful. But because they sound equally certain when they are wrong, you can never tell which answers to rely on. Such a friend is, in a practical sense, useless — not because they are often wrong, but because you cannot tell when they are wrong. Many machine learning systems behave exactly like this friend. The whole purpose of this discussion is to fix that: to give a model a principled, trustworthy way of saying "I am confident here" and "I am unsure here," and to make those statements mean something you can actually depend on.

The technical name for this goal is uncertainty quantification — literally, putting a number on how uncertain a prediction is. Let us start by understanding what "uncertainty" even means, because it turns out there are two quite different kinds, and telling them apart changes what you should do about them.

1.2 Two kinds of uncertainty

Suppose you are predicting how long it takes to drive to work. Some of your uncertainty comes from things that are genuinely random and unpredictable: whether you hit a red light, whether the car in front brakes suddenly, whether it rains. Even if you made this drive ten thousand times and recorded every detail, you could not predict the exact time on any given day, because the world itself contains randomness. This first kind of uncertainty is baked into the situation and no amount of extra information removes it.

The second kind is different. Suppose you have just moved to a new city and have driven to work only twice. Your uncertainty about the commute is now much larger — but not because the world got more random. It is larger because you personally do not yet know the route well. This uncertainty is about your own lack of knowledge, and it shrinks every time you make the drive and learn a little more.

These two kinds have names worth remembering.

The first — the irreducible randomness that lives in the world itself — is called aleatoric uncertainty. The word comes from the Latin alea, meaning a die, as in a rolled die. It is the uncertainty of the dice. To picture it with an engineering example: take several bars of steel that are, as far as anyone can measure, absolutely identical, and pull each one until it snaps. They will not all snap at exactly the same force. There are tiny variations in the metal's internal structure that no measurement captures. Testing a million more bars would tell you the range of snapping forces very precisely, but it would never let you predict the exact snapping force of the next bar. That leftover unpredictability is aleatoric — it belongs to the world.

The second kind — uncertainty that comes from the model not having seen enough, and that would shrink with more data — is called epistemic uncertainty. The word comes from the Greek episteme, meaning knowledge; epistemic uncertainty is uncertainty of knowledge. A model trained only on photographs taken in daylight is deeply epistemically uncertain about a night-time photograph, simply because it has never seen one. Show it many night-time photographs and that uncertainty falls away. Unlike aleatoric uncertainty, epistemic uncertainty is a property of the learner, not of the world, and it is reducible.

Why does the distinction matter? Because the two kinds call for opposite responses. If your uncertainty is aleatoric — genuine randomness you cannot remove — then collecting more data will not help, and the sensible engineering response is to design for the spread: build in a safety margin wide enough to accommodate the natural variation. If your uncertainty is epistemic — the model is in unfamiliar territory — then the sensible response is the opposite: either gather more data in that region, or have the system refuse to answer and hand the case to a human. Confusing the two leads to expensive mistakes: collecting mountains of data to fight randomness that will never yield, or blindly trusting a model in exactly the unfamiliar situations where it should have stayed silent.

Keep this pair in mind. Much of what follows is, at heart, machinery for detecting when a model has wandered into territory it does not understand — that is, machinery for exposing epistemic uncertainty — so that the system can react before a confident-sounding mistake reaches someone who trusts it.

1.3 Why modern AI is overconfident

We claimed above that most machine learning systems resemble the friend who is always certain. That is a strong claim, so let us see why it is true, and where the overconfidence comes from.

Modern AI systems — the deep neural networks behind image recognition, language models, and most recent advances — usually end with a step that produces a set of numbers looking very much like probabilities. Ask such a network to classify a photograph and it might report: cat 0.97, dog 0.02, everything else 0.01. It is tempting to read that 0.97 as "the model is 97% sure it's a cat," and to treat it as a confidence that can be blindly trusted.

Here is the uncomfortable fact: that number is usually not trustworthy as a probability. Networks of this kind routinely report 0.97, or even 0.999, for images they get completely wrong. This is not a flaw in one particular network that better engineering would fix. It is a predictable consequence of how these systems are built and trained, and it shows up almost everywhere.

To see why, we need to understand the final step, which is called the softmax. You do not need its formula yet; you need its purpose. A neural network's raw output is a list of unbounded scores, one per possible answer — say a score of 8.0 for "cat," 1.0 for "dog," and so on. These raw scores can be any size, positive or negative, and they do not add up to anything meaningful. The softmax is a piece of arithmetic that crunches this list into numbers that are all between 0 and 1 and that add up to exactly 1 — in other words, into something shaped like a set of probabilities. And that is the trap. The softmax guarantees the outputs will look like probabilities. It guarantees nothing whatsoever about whether they behave like probabilities — that is, whether a 0.9 really does correspond to being right nine times out of ten.

The overconfidence has a further, deeper reason in the way these networks are trained. Training nudges the network, over millions of examples, to make its answer on each training example as emphatically correct as possible — to push the score of the right answer higher and higher. Modern networks are enormous, with far more internal capacity than they strictly need to get the training examples right, and nothing stops them from driving their confidence on those examples all the way toward certainty. The result is a network that has, in effect, learned the habit of being emphatic, and it carries that habit into new situations where the emphasis is not warranted.

The practical lesson is this. Treat the confidence number from a raw neural network as a ranking signal (for deciding which of two answers the model considers more likely), but not as a genuine probability you can bet on. When the model says 0.9, you cannot conclude that it will be right 90% of the time. Establishing outputs you actually can bet on is the central problem of this discussion.

1.4 The gap between a number and a real chance
Let us make the previous section's warning concrete, because it is the exact gap the rest of the chapter exists to close.

Picture a model that looks at loan applications and outputs, for each one, a number meant to represent the chance the loan will be repaid. For one application it outputs 0.9. What should that 0.9 mean?

The meaning you want is a promise about the real world: "If you gathered up every application for which I said 0.9, then about 90 out of every 100 of them would actually be repaid." This is a clear, checkable promise. You could hold the model to it: collect a few hundred applications it scored near 0.9, wait to see how many were repaid, and count. If about 90% were repaid, the model kept its promise. This desirable property — that a stated confidence matches the real-world frequency of being correct — is called calibration, and we will return to it many times. A calibrated 0.9 is one you can act on.

The meaning you actually get from a raw network is far weaker. The 0.9 is just the softmax's reshaped score — a number that came out between 0 and 1 because the arithmetic forced it to, with nothing in the training process ensuring it matches any real-world frequency. When you go and check, you often find that among the applications the model scored 0.9, only 70% were actually repaid. The model said 0.9 but behaved like 0.7. Nobody lied; the number was simply never a real chance to begin with.

This mismatch between the confidence a model states and the frequency with which it is actually right is the single problem underneath everything in this discussion. There are broadly two ways to respond to it, and it is worth naming both now because they shape the whole subject.

The first response is to repair the numbers — to take the model's confidence scores and mathematically adjust them so that, after adjustment, a 0.9 really does correspond to being right about 90% of the time. This family of techniques is called calibration, and we will meet the main methods shortly.

The second response is more radical and is the true subject of this discussion. Instead of trying to fix a single confidence number, we change the shape of the answer entirely. Rather than have the model output one guess with a dubious confidence attached, we have it output a small set of possible answers — and we arrange things so that we can make an ironclad, mathematically guaranteed promise about that set: "the true answer is inside this set at least 90% of the time." This second approach is called conformal prediction, and in future chapters you will understand exactly how it delivers a promise that the raw softmax number never could.

1.5 What goes wrong when confidence is not trustworthy
Overconfidence is not merely untidy; it causes real harm, and the harm has a single recognizable shape. Here are four situations, described not as news stories but as mechanisms, so you can see the common pattern.

Consider first a medical model that helps triage patients. It was trained at one hospital, where a particular scan was ordered mostly for patients already suspected of being seriously ill. The model quietly learns to associate that scan with bad outcomes — not because the scan reveals the illness, but because of who tended to receive it. Move the model to a second hospital with different habits, and it fails. Worse, it fails confidently, reporting high certainty on exactly the patients a cautious system should have flagged for a doctor's second look.

Consider next a self-driving car's vision system, trained on data from one country's roads. On the roads of another country it meets unfamiliar signs, unfamiliar vehicles, unfamiliar light. It misreads them — and reports high confidence while doing so. The car's planning software, which was designed to slow down and ask for help whenever confidence drops, never receives the signal to do so, because the confidence never dropped.

Consider a credit model built during a calm economic period. When the economy lurches into a state unlike anything in its training data, the model keeps issuing crisp, confident predictions as though nothing has changed. Because it never signals that it has entered unfamiliar territory, the risk it is now badly wrong goes completely unnoticed until losses arrive.

Consider finally a language model asked a question whose answer does not appear in any document it was given. Instead of saying "I don't know," it produces a fluent, specific, confident answer that is simply invented. Fluency and confidence travel together in such models; correctness does not travel with them.

Look at the four together and the pattern is unmistakable: in every case, the system's confidence failed to fall when its actual competence fell. The model walked off the edge of what it knew, and its self-reported certainty never flinched. Every technique discussed here is, in one way or another, an attempt to make certainty flinch at the right moment.

1.6 Accuracy is not the same as honesty
There is a distinction here that is easy to blur, and blurring it causes real confusion, so we will state it plainly and then illustrate it.

A model can be accurate — right most of the time — while being dishonest about its confidence, in the sense that its stated certainty does not match how often it is actually right. And a model can be rather inaccurate while being scrupulously honest about its confidence. Accuracy and honesty-of-confidence are two separate qualities, and improving one does not automatically improve the other.

To make this vivid, imagine two weather forecasters. The first is right 95% of the time about whether it will rain, but every single morning she announces "99.9% chance" with the same booming certainty — including on the mornings she gets it wrong. She is highly accurate and badly miscalibrated: her stated confidence is a fixed performance, disconnected from reality. The second forecaster is right only 70% of the time, but he is careful — when he says "70% chance of rain," it does in fact rain on about 70% of such days. He is less accurate but perfectly honest, and for planning purposes his forecasts are often more useful, because you can trust the number he gives you.

The moral is that chasing accuracy alone does not give you a trustworthy system. A system can be accurate and still mislead you about when to trust it — which, as the triage and self-driving examples showed, is precisely the information that keeps people safe. Honesty about uncertainty is a distinct goal, and it must be pursued deliberately. It will not arrive on its own as a free side effect of making the model more accurate.

1.7 Why bother: the payoff of honest uncertainty
Before diving into methods, it is worth being concrete about what honest uncertainty actually buys you, because the benefits are larger and more immediate than newcomers usually expect. There are four, and they build naturally on the ideas we have just developed.

The first and often largest is the ability to know when to step aside. A system that can reliably tell which of its predictions are shaky can simply decline to answer those, and pass them to a human instead. On the predictions it does keep, its accuracy is then far higher, because the doubtful cases have been filtered out. This is called selective prediction, and the striking thing is that it requires no change to the underlying model at all — only an honest uncertainty signal wrapped around it. A model that is 85% accurate overall might, by declining its shakiest 15% of cases, be 97% accurate on the rest, with the hard cases safely in human hands.

The second is the ability to match the level of automation to the stakes. Not all decisions cost the same when they go wrong. A trustworthy uncertainty measure lets you set a strict bar for high-stakes decisions — automate only when the system is very sure — while allowing a relaxed bar for cheap, easily-reversed decisions. Without an honest uncertainty signal you are forced into a single blanket policy for everything, which is either recklessly loose for the important cases or needlessly strict for the trivial ones.

The third is the ability to satisfy the people who must sign off on the system. In regulated fields — aircraft, railways, cars, medical devices — approval increasingly demands evidence about how the system behaves under uncertainty, not just a headline accuracy figure. As we will see, conformal prediction offers something unusually powerful here: a guarantee that holds by mathematical proof rather than by empirical hope, which is exactly the kind of evidence a safety case is built from.

The fourth is the ability to notice trouble early. When the world shifts underneath a deployed model — new kinds of inputs start arriving — the model's uncertainty measures typically swell before anyone can measure a drop in accuracy, because measuring accuracy requires knowing the right answers, which often arrive weeks late or never. A rising tide of uncertainty is an early-warning system you can watch in real time, without waiting for ground truth. We will make this precise in Chapter 7.

1.8 Two roads: repairing numbers versus reshaping answers
We have twice mentioned that there are two responses to untrustworthy confidence. Now that the motivation is in place, let us lay the two side by side, because understanding how they differ is the doorway into the rest of the book.

The first road is calibration: keep the model's output in its familiar form — a single answer with a confidence number — but mathematically adjust the number so it becomes honest. The best-known method is almost absurdly simple. Recall that the softmax turns raw scores into probability-shaped numbers. It turns out that if you first divide all the raw scores by a single well-chosen number before applying the softmax, you can systematically deflate the model's overconfidence. That single number is called a temperature, and the method is temperature scaling. A large temperature squashes the scores together and makes the model more humble; a temperature of one leaves it unchanged. You find the right temperature by trying values on a batch of held-aside data and keeping whichever makes the confidence numbers best match the real frequency of being correct. Because dividing every score by the same number does not change which score is largest, temperature scaling never changes the model's actual answers — it only tempers their confidence. Two close relatives, Platt scaling and isotonic regression, do a similar job with slightly more flexibility; we will meet isotonic regression properly in Chapter 6.

To check whether calibration worked, we need a way to measure honesty, and the standard measure is worth understanding because it captures the idea directly. You take all the model's predictions and sort them into buckets by stated confidence — one bucket for predictions around 0.9, one for those around 0.8, and so on. Within each bucket you compare two things: the average confidence the model claimed, and the fraction of those predictions that were actually correct. If the model is honest, these two match in every bucket. The average size of the mismatch, across all buckets, is called the Expected Calibration Error, and a well-calibrated model has a small one. That is all the term means: the typical gap between claimed confidence and real accuracy.

The second road is conformal prediction, and it gives up on repairing the single number. Instead it changes what the model hands you. Rather than "the answer is B, confidence 0.72," a conformal system hands you a small set — "the answer is one of {B, D, F}" — together with a guarantee of a kind the first road can never offer: the true answer lies in sets like this at least 90% of the time, and this is provable, for any model, on any kind of data, from a finite amount of checking. The price is that the answer is now a set rather than a single label, which is less specific. The reward is that the promise attached to it is airtight.

It is worth being clear that these two roads are partners, not rivals. In fact a model whose numbers have first been repaired by calibration tends to produce smaller, more useful sets when conformal prediction is applied on top — better inputs make for tighter guarantees. The difference between them is one of ambition and of what can be promised, summarized here and then explained in the sentences that follow.

Reading the table across: calibration improves the quality of a number but can only promise approximate honesty, and only for data resembling what it was tuned on; when the world changes, it drifts out of honesty silently, giving no sign. Conformal prediction changes the form of the answer into a set, and in exchange can offer a promise that is exact and provable rather than approximate and hopeful. This contrast — an approximate fix to a number versus a provable guarantee about a set — is the pivot on which the entire book turns. Everything from Chapter 2 onward develops the second road.

A common misunderstanding, cleared up early "If temperature scaling makes my confidence numbers honest, why do I need conformal prediction at all?" Temperature scaling makes the confidence numbers honest on average, for data that looks like the batch you tuned it on. It makes no promise about any single prediction, no promise once the incoming data starts to look different, and no promise you can prove to a regulator. Conformal prediction provides a promise that is exact, provable, and holds for any model and any data — provided one condition, introduced in the next chapter, is met. The honest answer is to use both: calibrate the numbers first, then wrap conformal prediction around the result for a guarantee you can stand behind.

Key Takeaways — Chapter 1
Uncertainty comes in two flavors: aleatoric, the genuine randomness in the world that more data cannot remove, and epistemic, the model's own ignorance that more data can remove. The right response to each is opposite — absorb the first with a safety margin, and for the second, gather more data or step aside.

Modern neural networks are overconfident by construction. The softmax makes their outputs look like probabilities without making them behave like probabilities, and training instills a habit of emphatic certainty. A raw confidence number is a useful ranking signal but not a real chance you can bet on.

Accuracy and honesty-about-confidence are separate qualities. A model can be accurate yet misleading about when to trust it, which is exactly the information that keeps people safe.

Honest uncertainty pays off four ways: stepping aside on hard cases, matching automation to stakes, satisfying regulators with provable evidence, and catching trouble early before accuracy visibly drops.

There are two roads to trustworthy confidence. Calibration repairs the number but can only promise approximate honesty on familiar data. Conformal prediction reshapes the answer into a set and, in return, offers an exact and provable guarantee. Use both — but the second road is where this discussion is going.

(To be continued)

DEV Community: Karthik Amirapu

Uncertainty Quantification in AI (Part - 1)