A classic puzzle: a condition affects 1 in 1,000 people. A test catches 99% of real cases and wrongly flags 1% of healthy people. You test positive. How likely is it that you actually have the condition?
Most people guess around 99%. The real answer is under 10%. Here is the whole thing in a few lines of Python.
def posterior(prior, sensitivity, false_positive_rate):
true_pos = prior * sensitivity
false_pos = (1 - prior) * false_positive_rate
return true_pos / (true_pos + false_pos)
p = posterior(prior=0.001, sensitivity=0.99, false_positive_rate=0.01)
print(f"P(condition | positive) = {p:.1%}") # about 9.0%
# A second, independent positive test: use the first result as the new prior
p2 = posterior(prior=p, sensitivity=0.99, false_positive_rate=0.01)
print(f"After two positives = {p2:.1%}") # about 90.8%
Why it happens
Picture 100,000 people:
- 100 have the condition, and the test flags 99 of them.
- 99,900 are healthy, and the test still flags 1% of them: 999 people.
So out of 1,098 positives, only 99 are real. That is about 9%. The test is not bad. The condition is just rare, and false positives from the huge healthy group swamp the true ones.
The takeaway for engineers
The same trap shows up everywhere we build detectors: fraud alerts, anomaly monitors, spam filters, flaky test triage. A "99% accurate" alert on a rare event will mostly page you for nothing. Before trusting a signal, ask two questions:
- How rare is the thing I'm looking for (the base rate)?
- How often does the detector fire when nothing is wrong?
Then run the 15 lines. Your intuition will thank you.
What is the most surprising base-rate trap you have hit in production?
Top comments (0)