A model I was reviewing reported 0.98 ROC-AUC on a fraud dataset. Great number. Then it went live and roughly 95% of its alerts were false alarms. The metric wasn't wrong — it was answering a question nobody in production cared about. The plot that would have caught this up front is the precision-recall curve, so I built an interactive one you can poke at.
Live, computed in your browser: https://dev48v.infy.uk/ml/day31-precision-recall-curve.html
Drag a class-imbalance slider and watch the ROC curve stay gorgeous while the PR curve quietly collapses. That gap is the whole lesson.
Scores, not classes
A classifier doesn't hand you a label — it hands you a score. You turn scores into decisions by picking a threshold. Slide that threshold and every cell of the confusion matrix moves, which means two metrics move with it:
- Precision = TP / (TP + FP) — of everything you flagged, how much was actually right? This is the cost of a false alarm.
- Recall = TP / (TP + FN) — of everything that truly was positive, how much did you catch? This is the cost of a miss.
Lower the threshold and you catch more real positives (recall up) but drag in more junk (precision down). Raise it and you get pickier (precision up, recall down). There is no threshold that maximizes both. The precision-recall curve is just every one of those trade-off points plotted at once: recall on x, precision on y.
Building the curve is one pass
You don't test hundreds of arbitrary thresholds. The only thresholds that change anything are the scores themselves. So sort examples by score, walk down the list, and keep running TP and FP counts:
tp = fp = 0
points = [(0.0, 1.0)] # recall 0, precision 1
for label in labels_sorted_by_score_desc:
if label == 1: tp += 1
else: fp += 1
points.append((tp / n_pos, tp / (tp + fp))) # (recall, precision)
Recall sweeps from 0 to 1 in a single linear scan — no refitting, no rescanning.
Average Precision, and the baseline that actually matters
To compare models you want one number: Average Precision, the area under the PR curve. It's a weighted sum where each precision is weighted by the jump in recall since the last point — which, because recall only jumps when you rank a true positive correctly, works out to the mean precision measured at every positive. In scikit-learn that's two calls:
from sklearn.metrics import precision_recall_curve, average_precision_score
y_score = clf.predict_proba(X_test)[:, 1] # scores, not .predict()
prec, rec, thr = precision_recall_curve(y_test, y_score)
ap = average_precision_score(y_test, y_score)
baseline = y_test.mean() # <- the number everyone forgets
That last line is the trap people fall into. On a ROC curve the no-skill line is always the 0.5 diagonal. On a PR curve it is not 0.5 — it's the positive prevalence. If 1% of your data is positive, a coin-flip model already scores 0.01 precision, so "AP = 0.10" is a genuine 10x lift over chance. At balanced 50% prevalence that same 0.10 would be garbage. You cannot judge an AP without the baseline it sits above.
Why PR beats ROC on imbalance
Here's the crux, and it's a one-line observation. ROC plots recall against the false-positive rate, FPR = FP / (FP + TN). That denominator contains the true negatives. When negatives massively outnumber positives, TN is enormous, so even thousands of false positives produce a microscopic FPR and the ROC curve hugs the top-left corner. It looks incredible.
Precision is FP / (TP + FP). It never touches TN. Those same thousands of false positives sit right next to your handful of true positives and crush it. ROC hides the false alarms inside a sea of true negatives; PR puts them front and center. In the demo, cranking imbalance from 1:1 to 1:100 barely moves ROC-AUC (it stayed around 0.95 for me) while AP fell from ~0.96 to ~0.41. Same model, same ranking — one metric flatters it, the other tells the truth.
Pick a threshold on purpose
The curve describes the model; deployment needs one operating point, and it should come from cost, not from 0.5. Maximize F1 = 2PR/(P+R) if you want balance (the harmonic mean punishes lopsided precision/recall — 0.95 and 0.05 gives F1 ≈ 0.095, not 0.5). Or set a precision floor — "alerts must be right 80% of the time" — and take the highest recall on the curve that still clears it.
The habit worth building: on any problem where the positive class is rare, lead with the PR curve and AP, and treat a shiny ROC-AUC with suspicion until you've seen precision at the recall you actually need.
Play with the thresholds and the imbalance slider here: https://dev48v.infy.uk/ml/day31-precision-recall-curve.html
Top comments (0)