DEV Community: Karthik Amirapu

Uncertainty Quantification in AI through Conformal Prediction (Part — 2)

Karthik Amirapu — Fri, 31 Jul 2026 11:21:29 +0000

Chapter 2 — The Idea Behind Conformal Prediction
Chapter 1 closed with a promise: there is a way to make a model output a tiny set of answers, coupled with an unbreakable assurance that the truth lands inside such sets at least, say, 90% of the time — and this works for any model, on any sort of data. That promise seems too good, probably, to be true. This chapter convinces you that it is plausible by showing you the full idea in plain English. The next chapter translates the idea into precise mathematics. By the end of this chapter you will know why conformal prediction works, even though you can’t prove it yet.

2.1 Where the idea came from

The conformal prediction method has an odd pedigree. It evolved not out of ordinary statistics but out of a question about randomness itself, asked in the late 1990s by Vladimir Vovk, Alexander Gammerman, and Glenn Shafer, and gathered into a 2005 book of theirs. Vovk had studied under Kolmogorov, one of the founders of modern probability, and the seed of conformal prediction is a beautifully simple reframing that we will use throughout: instead of asking “what is the probability this new answer is correct?” — a hard question that seems to require knowing the true distribution of the data — ask instead “if I assumed this answer were correct, how strange or ordinary would that make my collection of data look?” This shift, from estimating a probability to measuring strangeness against data you already have, is the entire trick, and everything else is bookkeeping.

For roughly fifteen years the idea stayed inside a small research community. It broke into wide use around 2020 for three reasons that reinforced one another: the deep learning boom created an urgent need for trustworthy uncertainty, precisely because of the overconfidence described in Chapter 1; a clear and friendly tutorial by Anastasios Angelopoulos and Stephen Bates in 2021 translated the theory for working engineers; and ready-made software libraries appeared that made the method a few lines of code rather than a research project. We will discuss one of those libraries, MAPIE, later on.

2.2 The whole idea of Conformal Prediction

Here is conformal prediction from start to finish, with no mathematics. Read it slowly; everything afterward is elaboration.

You begin with a model that is already trained. You do not trust its confidence numbers — Chapter 1 explained why. But you also have a stack of labeled examples that the model has never seen during training: inputs for which you happen to know the correct answers. This stack , we will refer to it as the calibration set. Think of it as a fair exam the model must sit, where you already have the answer key.

First, you grade the model on this exam, but in a rather unusual way. For each example in the calibration set, you ask a single question: how badly did the model do on this one? You boil that down to one number, where a bigger number means the model did worse. This “how badly did it do” number is called the nonconformity score — nonconformity because it measures how poorly the example conforms to what the model expected. For a classifier, a natural such score is one minus the probability the model assigned to the correct answer: if the model gave the true answer a probability of 0.95, the model did well, and the score is a small 0.05; if it gave the true answer only 0.30, the model did poorly, and the score is a large 0.70. You compute this score for every example in the calibration set, and now you hold a whole pile of “how badly the model tends to do” numbers.

Second, you find the value that these scores rarely exceed. Concretely, if you want a 90% guarantee, you find the number below which 90% of your calibration scores fall — the point where only the worst 10% of scores lie above it. Call this cutoff value the threshold. It represents, in effect, the boundary of “normal badness”: scores below it are the kind of mistake the model makes routinely, and scores above it are unusually bad. (The next section explains how to find such a cutoff; for now, just hold the idea of “the value that 90% of scores fall below.”)

Third, a new input arrives and you must make a prediction. Here is the clever move. You consider every possible answer the input could have, and for each candidate answer you compute what its nonconformity score would be if that answer were the true one. Then you keep every candidate answer whose score falls below the threshold — every answer that would represent no more than “normal badness” — and you throw away the rest. The answers you keep are your prediction set. That set is your output: not a single guess, but the collection of all answers that are plausible given how the model normally behaves.

Now, why does this give a 90% guarantee? The reasoning is short and, once you see it, hard to unsee. The new input is, we will assume, just another example drawn from the same source as the calibration examples — one more student sitting the same exam. So its nonconformity score is just one more number of the same kind as the pile we already have. Ask: how often does one more number from the same pile land below the 90% threshold? By the very meaning of “the value 90% of scores fall below,” it lands below about 90% of the time. And whenever the true answer’s score lands below the threshold, the true answer was, by construction, kept in the set. Therefore the true answer sits inside the prediction set about 90% of the time. That is the guarantee, and notice what the argument never once needed: it never needed the model to be good, and it never needed the data to follow any particular bell curve or formula. It needed only that the new example is of the same kind as the calibration examples.

2.3 What a quantile is, and how to find the threshold
The second step above asked for “the value 90% of scores fall below.” That value has a name — a quantile — and because quantiles appear on every page from here on, we pause to make the idea completely solid. If you are already comfortable with percentiles, this is the same thing under a more formal name, and you can skim.

Suppose you have the test scores of 100 students, and you want the score that separates the bottom 90% from the top 10%. You line all 100 scores up from smallest to largest and walk 90% of the way along the line. The score you are standing next to is the 90th percentile, or equivalently the 0.9 quantile: 90% of students scored at or below it, 10% scored above. A quantile, then, is nothing more mysterious than a cutoff at a chosen position in a sorted list. The 0.5 quantile is the middle value, better known as the median. The 0.9 quantile is the value nine-tenths of the way up. The general recipe is: sort the numbers, then pick the one at the position corresponding to your chosen fraction.

For conformal prediction, the numbers we sort are the calibration set’s nonconformity scores, and the fraction we choose corresponds to the guarantee we want. Want to be right 90% of the time? Take roughly the 0.9 quantile of the scores as your threshold. There is one small but genuinely important adjustment to exactly which position we pick — it involves counting the new test example as well, and rounding up rather than down — but that adjustment is a detail of Chapter 3, and skipping it now costs nothing in understanding. The essential picture is simply this: sort the “how badly did the model do” scores, and take the value near the top below which your target fraction of them fall. That value is the threshold, and it is the only thing you need to store to start making guaranteed predictions.

2.4 The bargain at the heart of the method

Step back and notice the strange and wonderful bargain we have struck. The guarantee — the 90% coverage — does not depend on the model being any good. A hopeless model and a brilliant model both achieve exactly 90% coverage. This raises an obvious worry: if a hopeless model also hits 90%, is the guarantee worthless?

The answer is the single most important intuition in the whole subject — the model’s quality does not change whether the guarantee holds — it changes the size of the sets needed to hold it.

Picture the extreme. A completely useless model, one that has learned nothing, can still be wrapped in conformal prediction and still achieve 90% coverage — but it does so by returning enormous sets. If it must include almost every possible answer to be sure the truth is among them 90% of the time, then it does exactly that, and its “prediction sets” are so large as to be useless. Meanwhile a genuinely good model achieves the same 90% coverage with small, sharp sets — often just a single answer — because its nonconformity scores separate right answers from wrong ones cleanly, and only a few candidates ever fall below the threshold.

So conformal prediction performs a kind of translation. It takes the quality of your model, which is hard to state as a promise, and converts it into the size of the prediction sets, while holding the coverage fixed at whatever you asked for. Validity — the guarantee itself — is free and automatic. Efficiency — small, useful sets — is what a good model earns. Whenever you find yourself worrying “but surely a bad model breaks the guarantee,” remember: it does not break the guarantee, it just makes the sets fat. This bargain is what lets the guarantee be honest without being empty.

2.5 The one assumption: exchangeability

We have leaned twice now on a phrase: “assuming the new example is of the same kind as the calibration examples.” It is time to make that assumption precise, because it is the only assumption conformal prediction makes, and — as we will see repeatedly — it is also the only thing that can make conformal prediction fail. Everything rests on it, so it deserves careful attention.

The assumption is called exchangeability, and the plain-language version is this: the order in which your examples arrived carries no information. Imagine writing each of your labeled examples on a card, shuffling all the cards together — the calibration examples and the new test example alike — and dealing them out in a row. Exchangeability says that no possible ordering of the cards is any more likely than any other. There is nothing special about which example happened to come “last.” If you cannot tell, from the pattern in the data, which card was the test point, then the examples are exchangeable.

Why is this exactly the assumption we need? Look back at the guarantee in Section 2.2. Its entire force came from treating the new example’s score as “just one more number from the same pile,” equally likely to land in any position among the sorted scores. That is precisely what exchangeability grants: if any ordering is as likely as any other, then the new score is equally likely to be the smallest, the largest, or anywhere in between, and the “lands below the 90% threshold about 90% of the time” reasoning goes through. Remove exchangeability and that reasoning collapses.

It helps to know that exchangeability is a mild assumption — milder than the “independent and identically distributed” condition often assumed in statistics, which you may or may not have met. You do not need the details of that comparison; the takeaway is that exchangeability asks for less, and is therefore satisfied more often, which is part of why the method is so widely applicable. A plain random split of a fixed dataset into training, calibration, and test portions is exchangeable, and that covers a great many ordinary machine learning projects.

But — and this is the warning that will echo through every later chapter — exchangeability fails in several extremely common situations, and when it fails the guarantee silently stops being true. The most important failures are worth previewing now so you can start recognizing them.

The clearest failure is time. Data that arrives in time order — stock prices, daily electricity demand, sensor readings — is almost never exchangeable, because the order obviously carries information: yesterday’s value tells you something about today’s, and last winter is not interchangeable with next summer. Shuffle a temperature record and you have destroyed exactly the seasonal pattern that made it predictable. For time-ordered data, plain conformal prediction is not valid, and Chapters 7 and 9 develop the repairs.

A second failure is a change in the world between calibration and use. If your calibration examples came from Hospital A and your model is deployed at Hospital B, whose patients differ, then the test examples are not interchangeable with the calibration examples — you could tell them apart — and the guarantee does not carry over. This situation, called distribution shift, is so important that Chapter 7 is largely about it.

A third, quieter failure is accidental contamination: using the same data both to tune the model and to calibrate it. If the calibration set influenced the model in any way — helping choose its settings, or deciding when to stop training — then it is no longer a fair, unseen exam, and the exchangeability between calibration and test is broken in a way that is easy to miss and that quietly inflates your apparent guarantee. Keeping the calibration set truly untouched is a discipline we will insist on.

The single most valuable habit you can build, before ever deploying conformal prediction, is to pause and ask: are my calibration examples and my real incoming examples genuinely interchangeable? If yes, the guarantee is yours. If no, you must either fix the situation or use one of the specialized methods of later chapters. Exchangeability is the foundation; check it before you build.

A common misunderstanding “Conformal prediction makes no assumptions — it works on any data.” It makes exactly one assumption: exchangeability. That is a weak assumption, satisfied by ordinary random data splits, but it is not no assumption. It is broken by time-ordered data, by deployment on a population different from the calibration one, and by accidentally reusing calibration data during tuning — which, unfortunately, are three of the most common situations in real projects. The method is assumption-light, not assumption-free.

2.6 A guarantee that holds even with little data
There is one more property of the guarantee that sets conformal prediction apart, and it is easy to appreciate once you know what to contrast it with.

Many promises in statistics and machine learning are of the form “this becomes true as you collect more and more data” — they hold in the limit of infinite data, and are only approximately true for the finite amount of data you actually have. That is cold comfort when your calibration set has 400 examples, because the promise does not quite tell you what happens at 400.

Conformal prediction’s guarantee is not of that kind. It is exact and it holds for any size of calibration set, however small — this property is called being a finite-sample guarantee, meaning it is true for the finite sample you actually possess, not merely in some infinite ideal. If you ask for 90% coverage and your assumptions hold, you get at least 90% coverage whether your calibration set has 50 examples or 50,000. There is no approximation and no waiting for the numbers to settle down.

This does not mean the size of the calibration set is irrelevant — it matters, but in a subtler way than you might expect, and we will quantify it in Chapter 3. The short version, to hold until then: with a small calibration set the guarantee is still true on average, but the coverage you get on any particular occasion can wobble a bit above or below your target, because a small exam is a noisier measure of the model than a large one. With around a thousand calibration examples that wobble becomes small, and the coverage you actually get sits reliably close to the coverage you asked for. More data does not make the guarantee “more true”; it makes the guarantee less variable.

2.7 On average versus every single time

Everything so far has spoken of coverage as an overall rate: “the truth is in the set 90% of the time.” It is important — and it is a frequent source of confusion — to understand exactly what that rate is averaged over, because the guarantee is slightly weaker than a first reading suggests, in a way that occasionally matters a great deal.

The guarantee conformal prediction actually delivers is called marginal coverage, and the word “marginal” here means “averaged over all inputs.” The promise is: across the whole population of inputs you might encounter, taken together, 90% of the prediction sets contain the truth. This is an average over the whole mix of easy and hard cases.

What the guarantee does not promise is that every individual type of input is covered 90% of the time. This stronger property — coverage holding not just overall but within every subgroup of inputs — is called conditional coverage, and here is the uncomfortable fact: for a method that assumes nothing about the data distribution, true conditional coverage is mathematically impossible to guarantee. This is not a shortcoming of present-day techniques that better ones will fix; it is a proven impossibility. You cannot, in general, promise 90% coverage on every kind of input at once without occasionally producing infinitely large sets.

Why does this matter in practice? Because an average can hide a lopsided reality. A system with an honest 90% marginal coverage might be achieving 99% coverage on the common, easy inputs and only 60% coverage on the rare, hard ones — and average out to a perfectly truthful 90%. The system is doing exactly what it promised; it is just that the promise was about the average, and the average was not the thing you cared about. If the rare, hard inputs are the medically dangerous ones, a truthful 90% average is cold comfort.

There is a practical middle path, used constantly in real systems, and it is worth naming now even though we develop it later. If you care about coverage within specific groups — say, you insist on 90% coverage separately for each disease a medical model might diagnose, not merely 90% overall — you can simply run the whole calibration procedure separately within each group, computing a different threshold for each. This gives you a guarantee inside each group and is the standard fix when a single overall average is not good enough. It is called group-conditional coverage, and it is the achievable compromise between the easy-but-sometimes-lopsided marginal guarantee and the impossible perfect conditional one.

The habit to build: the guarantee you get by default is an average. Always ask whether an average is what your problem actually needs, and if it is not, calibrate within the groups that matter and check each one.

A common misunderstanding “90% coverage means each individual prediction I make has a 90% chance of being correct.” Not quite. It means that across many predictions, 90% of the sets contain the truth. Any single set either contains the truth or does not — there is no “90% chance” hiding inside one particular set. Coverage is a property of the long-run behavior of the procedure, not a property of any one output. And that 90% is an average across all input types, which, as just discussed, can hide very different rates for different kinds of input.

2.8 Why the method spread so quickly
We can now collect the reasons conformal prediction went from a niche idea to a standard tool in a few years, because each reason is really a property we have already met, seen from the point of view of an engineer deciding whether to adopt it.

It wraps around any model without needing to look inside it — a deep network, a decision tree, even a paid service you can only send inputs to and receive outputs from. All it ever uses is the model’s outputs on the calibration set, never its internals. It requires no retraining: the model already running in production stays exactly as it is, and conformal prediction is bolted on afterward. It is cheap, costing essentially one pass over the calibration set and a single sort, after which each new prediction costs almost nothing. Its guarantee is a theorem, not a hopeful observation from a test run, which in a regulated industry is a categorically stronger kind of evidence. And its output is understandable by a human expert: “the diagnosis is one of these three, and statements like this are correct 95% of the time” is something a doctor can act on, in a way that “the softmax value was 0.87” never was.

Against these strengths stand the costs we have been honest about throughout: exchangeability must genuinely hold; the guarantee is an average rather than a per-input promise; you must set aside some labeled data for calibration; and a weak underlying model yields large, uninformative sets. None of these costs is hidden, and each has a known remedy or workaround developed in the chapters ahead. That combination — real strengths, honestly-stated costs, known remedies — is what earned the method its place.

Key Takeaways — Chapter 2
Conformal prediction reframes the hard question “how likely is this answer correct?” into the easier question “how strange would this answer make my data look?” — measured against a calibration set the model has never seen.

The method has three steps: grade the model on the calibration set to get a “how badly did it do” score for each example; find the threshold below which your target fraction of those scores fall; and for a new input, keep every candidate answer whose score falls below that threshold. The kept answers are the prediction set.

A quantile is just a cutoff at a chosen position in a sorted list — the 0.9 quantile is the value 90% of the numbers fall below. The threshold is a quantile of the calibration scores.

The central bargain: model quality does not affect whether the 90% guarantee holds, only the size of the sets needed to hold it. A bad model gives valid but huge sets; a good model gives valid and small ones.

The method’s one assumption is exchangeability — that the order of the data carries no information. It is mild but real, and it is broken by time-ordered data, by deployment on a shifted population, and by accidentally reusing calibration data in tuning. Check it before deploying.

The guarantee is exact for any calibration set size (finite-sample), but small calibration sets make the achieved coverage wobble around the target. It guarantees coverage on average across all inputs (marginal coverage); guaranteeing it for every subgroup (conditional coverage) is impossible in general, and the practical fix is to calibrate separately within the groups you care about.

Uncertainty Quantification in AI (Part - 1)

Karthik Amirapu — Tue, 28 Jul 2026 09:11:09 +0000

The following is series of posts on Uncertainity quantification in AI systems and discussion on some of the statistical methods to overcome

Chapter 1 — Why AI Needs Uncertainty

1.1 The problem in one sentence

Note- AI confidently hallucinating the spelling of 'trustworthy' in the image above - kind of proving the point of this article, it cant be trustworthy (pun intended) all the time.

Every prediction a machine learning system makes is a guess about something it has never seen before. A model that reads a chest X-ray and says "pneumonia" is guessing. A model that predicts tomorrow's electricity demand is guessing. This is not a criticism — guessing from patterns in past data is exactly what these systems are built to do, and they often do it remarkably well. The difficulty is not that they guess. The difficulty is that most of them guess without ever telling us how much to trust the guess.

Imagine a friend who answers every question you ask with total confidence, whether they know the answer or not. When they happen to be right, they are useful. But because they sound equally certain when they are wrong, you can never tell which answers to rely on. Such a friend is, in a practical sense, useless — not because they are often wrong, but because you cannot tell when they are wrong. Many machine learning systems behave exactly like this friend. The whole purpose of this discussion is to fix that: to give a model a principled, trustworthy way of saying "I am confident here" and "I am unsure here," and to make those statements mean something you can actually depend on.

The technical name for this goal is uncertainty quantification — literally, putting a number on how uncertain a prediction is. Let us start by understanding what "uncertainty" even means, because it turns out there are two quite different kinds, and telling them apart changes what you should do about them.

1.2 Two kinds of uncertainty

Suppose you are predicting how long it takes to drive to work. Some of your uncertainty comes from things that are genuinely random and unpredictable: whether you hit a red light, whether the car in front brakes suddenly, whether it rains. Even if you made this drive ten thousand times and recorded every detail, you could not predict the exact time on any given day, because the world itself contains randomness. This first kind of uncertainty is baked into the situation and no amount of extra information removes it.

The second kind is different. Suppose you have just moved to a new city and have driven to work only twice. Your uncertainty about the commute is now much larger — but not because the world got more random. It is larger because you personally do not yet know the route well. This uncertainty is about your own lack of knowledge, and it shrinks every time you make the drive and learn a little more.

These two kinds have names worth remembering.

The first — the irreducible randomness that lives in the world itself — is called aleatoric uncertainty. The word comes from the Latin alea, meaning a die, as in a rolled die. It is the uncertainty of the dice. To picture it with an engineering example: take several bars of steel that are, as far as anyone can measure, absolutely identical, and pull each one until it snaps. They will not all snap at exactly the same force. There are tiny variations in the metal's internal structure that no measurement captures. Testing a million more bars would tell you the range of snapping forces very precisely, but it would never let you predict the exact snapping force of the next bar. That leftover unpredictability is aleatoric — it belongs to the world.

The second kind — uncertainty that comes from the model not having seen enough, and that would shrink with more data — is called epistemic uncertainty. The word comes from the Greek episteme, meaning knowledge; epistemic uncertainty is uncertainty of knowledge. A model trained only on photographs taken in daylight is deeply epistemically uncertain about a night-time photograph, simply because it has never seen one. Show it many night-time photographs and that uncertainty falls away. Unlike aleatoric uncertainty, epistemic uncertainty is a property of the learner, not of the world, and it is reducible.

Why does the distinction matter? Because the two kinds call for opposite responses. If your uncertainty is aleatoric — genuine randomness you cannot remove — then collecting more data will not help, and the sensible engineering response is to design for the spread: build in a safety margin wide enough to accommodate the natural variation. If your uncertainty is epistemic — the model is in unfamiliar territory — then the sensible response is the opposite: either gather more data in that region, or have the system refuse to answer and hand the case to a human. Confusing the two leads to expensive mistakes: collecting mountains of data to fight randomness that will never yield, or blindly trusting a model in exactly the unfamiliar situations where it should have stayed silent.

Keep this pair in mind. Much of what follows is, at heart, machinery for detecting when a model has wandered into territory it does not understand — that is, machinery for exposing epistemic uncertainty — so that the system can react before a confident-sounding mistake reaches someone who trusts it.

1.3 Why modern AI is overconfident

We claimed above that most machine learning systems resemble the friend who is always certain. That is a strong claim, so let us see why it is true, and where the overconfidence comes from.

Modern AI systems — the deep neural networks behind image recognition, language models, and most recent advances — usually end with a step that produces a set of numbers looking very much like probabilities. Ask such a network to classify a photograph and it might report: cat 0.97, dog 0.02, everything else 0.01. It is tempting to read that 0.97 as "the model is 97% sure it's a cat," and to treat it as a confidence that can be blindly trusted.

Here is the uncomfortable fact: that number is usually not trustworthy as a probability. Networks of this kind routinely report 0.97, or even 0.999, for images they get completely wrong. This is not a flaw in one particular network that better engineering would fix. It is a predictable consequence of how these systems are built and trained, and it shows up almost everywhere.

To see why, we need to understand the final step, which is called the softmax. You do not need its formula yet; you need its purpose. A neural network's raw output is a list of unbounded scores, one per possible answer — say a score of 8.0 for "cat," 1.0 for "dog," and so on. These raw scores can be any size, positive or negative, and they do not add up to anything meaningful. The softmax is a piece of arithmetic that crunches this list into numbers that are all between 0 and 1 and that add up to exactly 1 — in other words, into something shaped like a set of probabilities. And that is the trap. The softmax guarantees the outputs will look like probabilities. It guarantees nothing whatsoever about whether they behave like probabilities — that is, whether a 0.9 really does correspond to being right nine times out of ten.

The overconfidence has a further, deeper reason in the way these networks are trained. Training nudges the network, over millions of examples, to make its answer on each training example as emphatically correct as possible — to push the score of the right answer higher and higher. Modern networks are enormous, with far more internal capacity than they strictly need to get the training examples right, and nothing stops them from driving their confidence on those examples all the way toward certainty. The result is a network that has, in effect, learned the habit of being emphatic, and it carries that habit into new situations where the emphasis is not warranted.

The practical lesson is this. Treat the confidence number from a raw neural network as a ranking signal (for deciding which of two answers the model considers more likely), but not as a genuine probability you can bet on. When the model says 0.9, you cannot conclude that it will be right 90% of the time. Establishing outputs you actually can bet on is the central problem of this discussion.

1.4 The gap between a number and a real chance
Let us make the previous section's warning concrete, because it is the exact gap the rest of the chapter exists to close.

Picture a model that looks at loan applications and outputs, for each one, a number meant to represent the chance the loan will be repaid. For one application it outputs 0.9. What should that 0.9 mean?

The meaning you want is a promise about the real world: "If you gathered up every application for which I said 0.9, then about 90 out of every 100 of them would actually be repaid." This is a clear, checkable promise. You could hold the model to it: collect a few hundred applications it scored near 0.9, wait to see how many were repaid, and count. If about 90% were repaid, the model kept its promise. This desirable property — that a stated confidence matches the real-world frequency of being correct — is called calibration, and we will return to it many times. A calibrated 0.9 is one you can act on.

The meaning you actually get from a raw network is far weaker. The 0.9 is just the softmax's reshaped score — a number that came out between 0 and 1 because the arithmetic forced it to, with nothing in the training process ensuring it matches any real-world frequency. When you go and check, you often find that among the applications the model scored 0.9, only 70% were actually repaid. The model said 0.9 but behaved like 0.7. Nobody lied; the number was simply never a real chance to begin with.

This mismatch between the confidence a model states and the frequency with which it is actually right is the single problem underneath everything in this discussion. There are broadly two ways to respond to it, and it is worth naming both now because they shape the whole subject.

The first response is to repair the numbers — to take the model's confidence scores and mathematically adjust them so that, after adjustment, a 0.9 really does correspond to being right about 90% of the time. This family of techniques is called calibration, and we will meet the main methods shortly.

The second response is more radical and is the true subject of this discussion. Instead of trying to fix a single confidence number, we change the shape of the answer entirely. Rather than have the model output one guess with a dubious confidence attached, we have it output a small set of possible answers — and we arrange things so that we can make an ironclad, mathematically guaranteed promise about that set: "the true answer is inside this set at least 90% of the time." This second approach is called conformal prediction, and in future chapters you will understand exactly how it delivers a promise that the raw softmax number never could.

1.5 What goes wrong when confidence is not trustworthy
Overconfidence is not merely untidy; it causes real harm, and the harm has a single recognizable shape. Here are four situations, described not as news stories but as mechanisms, so you can see the common pattern.

Consider first a medical model that helps triage patients. It was trained at one hospital, where a particular scan was ordered mostly for patients already suspected of being seriously ill. The model quietly learns to associate that scan with bad outcomes — not because the scan reveals the illness, but because of who tended to receive it. Move the model to a second hospital with different habits, and it fails. Worse, it fails confidently, reporting high certainty on exactly the patients a cautious system should have flagged for a doctor's second look.

Consider next a self-driving car's vision system, trained on data from one country's roads. On the roads of another country it meets unfamiliar signs, unfamiliar vehicles, unfamiliar light. It misreads them — and reports high confidence while doing so. The car's planning software, which was designed to slow down and ask for help whenever confidence drops, never receives the signal to do so, because the confidence never dropped.

Consider a credit model built during a calm economic period. When the economy lurches into a state unlike anything in its training data, the model keeps issuing crisp, confident predictions as though nothing has changed. Because it never signals that it has entered unfamiliar territory, the risk it is now badly wrong goes completely unnoticed until losses arrive.

Consider finally a language model asked a question whose answer does not appear in any document it was given. Instead of saying "I don't know," it produces a fluent, specific, confident answer that is simply invented. Fluency and confidence travel together in such models; correctness does not travel with them.

Look at the four together and the pattern is unmistakable: in every case, the system's confidence failed to fall when its actual competence fell. The model walked off the edge of what it knew, and its self-reported certainty never flinched. Every technique discussed here is, in one way or another, an attempt to make certainty flinch at the right moment.

1.6 Accuracy is not the same as honesty
There is a distinction here that is easy to blur, and blurring it causes real confusion, so we will state it plainly and then illustrate it.

A model can be accurate — right most of the time — while being dishonest about its confidence, in the sense that its stated certainty does not match how often it is actually right. And a model can be rather inaccurate while being scrupulously honest about its confidence. Accuracy and honesty-of-confidence are two separate qualities, and improving one does not automatically improve the other.

To make this vivid, imagine two weather forecasters. The first is right 95% of the time about whether it will rain, but every single morning she announces "99.9% chance" with the same booming certainty — including on the mornings she gets it wrong. She is highly accurate and badly miscalibrated: her stated confidence is a fixed performance, disconnected from reality. The second forecaster is right only 70% of the time, but he is careful — when he says "70% chance of rain," it does in fact rain on about 70% of such days. He is less accurate but perfectly honest, and for planning purposes his forecasts are often more useful, because you can trust the number he gives you.

The moral is that chasing accuracy alone does not give you a trustworthy system. A system can be accurate and still mislead you about when to trust it — which, as the triage and self-driving examples showed, is precisely the information that keeps people safe. Honesty about uncertainty is a distinct goal, and it must be pursued deliberately. It will not arrive on its own as a free side effect of making the model more accurate.

1.7 Why bother: the payoff of honest uncertainty
Before diving into methods, it is worth being concrete about what honest uncertainty actually buys you, because the benefits are larger and more immediate than newcomers usually expect. There are four, and they build naturally on the ideas we have just developed.

The first and often largest is the ability to know when to step aside. A system that can reliably tell which of its predictions are shaky can simply decline to answer those, and pass them to a human instead. On the predictions it does keep, its accuracy is then far higher, because the doubtful cases have been filtered out. This is called selective prediction, and the striking thing is that it requires no change to the underlying model at all — only an honest uncertainty signal wrapped around it. A model that is 85% accurate overall might, by declining its shakiest 15% of cases, be 97% accurate on the rest, with the hard cases safely in human hands.

The second is the ability to match the level of automation to the stakes. Not all decisions cost the same when they go wrong. A trustworthy uncertainty measure lets you set a strict bar for high-stakes decisions — automate only when the system is very sure — while allowing a relaxed bar for cheap, easily-reversed decisions. Without an honest uncertainty signal you are forced into a single blanket policy for everything, which is either recklessly loose for the important cases or needlessly strict for the trivial ones.

The third is the ability to satisfy the people who must sign off on the system. In regulated fields — aircraft, railways, cars, medical devices — approval increasingly demands evidence about how the system behaves under uncertainty, not just a headline accuracy figure. As we will see, conformal prediction offers something unusually powerful here: a guarantee that holds by mathematical proof rather than by empirical hope, which is exactly the kind of evidence a safety case is built from.

The fourth is the ability to notice trouble early. When the world shifts underneath a deployed model — new kinds of inputs start arriving — the model's uncertainty measures typically swell before anyone can measure a drop in accuracy, because measuring accuracy requires knowing the right answers, which often arrive weeks late or never. A rising tide of uncertainty is an early-warning system you can watch in real time, without waiting for ground truth. We will make this precise in Chapter 7.

1.8 Two roads: repairing numbers versus reshaping answers
We have twice mentioned that there are two responses to untrustworthy confidence. Now that the motivation is in place, let us lay the two side by side, because understanding how they differ is the doorway into the rest of the book.

The first road is calibration: keep the model's output in its familiar form — a single answer with a confidence number — but mathematically adjust the number so it becomes honest. The best-known method is almost absurdly simple. Recall that the softmax turns raw scores into probability-shaped numbers. It turns out that if you first divide all the raw scores by a single well-chosen number before applying the softmax, you can systematically deflate the model's overconfidence. That single number is called a temperature, and the method is temperature scaling. A large temperature squashes the scores together and makes the model more humble; a temperature of one leaves it unchanged. You find the right temperature by trying values on a batch of held-aside data and keeping whichever makes the confidence numbers best match the real frequency of being correct. Because dividing every score by the same number does not change which score is largest, temperature scaling never changes the model's actual answers — it only tempers their confidence. Two close relatives, Platt scaling and isotonic regression, do a similar job with slightly more flexibility; we will meet isotonic regression properly in Chapter 6.

To check whether calibration worked, we need a way to measure honesty, and the standard measure is worth understanding because it captures the idea directly. You take all the model's predictions and sort them into buckets by stated confidence — one bucket for predictions around 0.9, one for those around 0.8, and so on. Within each bucket you compare two things: the average confidence the model claimed, and the fraction of those predictions that were actually correct. If the model is honest, these two match in every bucket. The average size of the mismatch, across all buckets, is called the Expected Calibration Error, and a well-calibrated model has a small one. That is all the term means: the typical gap between claimed confidence and real accuracy.

The second road is conformal prediction, and it gives up on repairing the single number. Instead it changes what the model hands you. Rather than "the answer is B, confidence 0.72," a conformal system hands you a small set — "the answer is one of {B, D, F}" — together with a guarantee of a kind the first road can never offer: the true answer lies in sets like this at least 90% of the time, and this is provable, for any model, on any kind of data, from a finite amount of checking. The price is that the answer is now a set rather than a single label, which is less specific. The reward is that the promise attached to it is airtight.

It is worth being clear that these two roads are partners, not rivals. In fact a model whose numbers have first been repaired by calibration tends to produce smaller, more useful sets when conformal prediction is applied on top — better inputs make for tighter guarantees. The difference between them is one of ambition and of what can be promised, summarized here and then explained in the sentences that follow.

Reading the table across: calibration improves the quality of a number but can only promise approximate honesty, and only for data resembling what it was tuned on; when the world changes, it drifts out of honesty silently, giving no sign. Conformal prediction changes the form of the answer into a set, and in exchange can offer a promise that is exact and provable rather than approximate and hopeful. This contrast — an approximate fix to a number versus a provable guarantee about a set — is the pivot on which the entire book turns. Everything from Chapter 2 onward develops the second road.

A common misunderstanding, cleared up early "If temperature scaling makes my confidence numbers honest, why do I need conformal prediction at all?" Temperature scaling makes the confidence numbers honest on average, for data that looks like the batch you tuned it on. It makes no promise about any single prediction, no promise once the incoming data starts to look different, and no promise you can prove to a regulator. Conformal prediction provides a promise that is exact, provable, and holds for any model and any data — provided one condition, introduced in the next chapter, is met. The honest answer is to use both: calibrate the numbers first, then wrap conformal prediction around the result for a guarantee you can stand behind.

Key Takeaways — Chapter 1
Uncertainty comes in two flavors: aleatoric, the genuine randomness in the world that more data cannot remove, and epistemic, the model's own ignorance that more data can remove. The right response to each is opposite — absorb the first with a safety margin, and for the second, gather more data or step aside.

Modern neural networks are overconfident by construction. The softmax makes their outputs look like probabilities without making them behave like probabilities, and training instills a habit of emphatic certainty. A raw confidence number is a useful ranking signal but not a real chance you can bet on.

Accuracy and honesty-about-confidence are separate qualities. A model can be accurate yet misleading about when to trust it, which is exactly the information that keeps people safe.

Honest uncertainty pays off four ways: stepping aside on hard cases, matching automation to stakes, satisfying regulators with provable evidence, and catching trouble early before accuracy visibly drops.

There are two roads to trustworthy confidence. Calibration repairs the number but can only promise approximate honesty on familiar data. Conformal prediction reshapes the answer into a set and, in return, offers an exact and provable guarantee. Use both — but the second road is where this discussion is going.

(To be continued)