Your logistic regression model reports 99% accuracy.
Your team celebrates.
Then the fraud department tells you it caught almost none of the fraudulent transactions.
What happened?
Nothing is necessarily wrong with the logistic regression algorithm. The problem is probably how you're evaluating it.
Binary classification models don't just need a model score. They need the - right evaluation strategy - one that reflects what happens when the model makes a wrong prediction.
In this article, we'll walk through the core classification metrics you should know when evaluating a binary logistic regression model:
- Accuracy
- Precision
- Recall
- False positives and false negatives
- Threshold selection
- ROC curves
- ROC-AUC
We'll also use scikit-learn to calculate everything with real Python code.
First: Logistic Regression Doesn't Actually Predict "Yes" or "No"
Despite its name, logistic regression produces a probability.
For example, imagine we're predicting whether a credit-card transaction is fraudulent:
Transaction A → 0.03
Transaction B → 0.17
Transaction C → 0.82
Transaction D → 0.94
These numbers represent the model's estimated probability that each transaction belongs to the positive class - in this case, fraud.
To turn those probabilities into binary predictions, we choose a classification threshold.
The commonly used default is:
probability >= 0.50 → positive
probability < 0.50 → negative
So:
0.82 → Fraud
0.94 → Fraud
0.17 → Not Fraud
0.03 → Not Fraud
And this is where things get interesting.
The Flaw of the 0.5 Threshold
The 50% threshold feels reasonable.
After all, if the model says there's a greater than 50% chance of fraud, shouldn't we call it fraud?
Not necessarily.
Consider a medical screening system where missing a genuinely sick patient could have serious consequences.
Suppose the model outputs:
Patient A → 0.42 probability of disease
Patient B → 0.73 probability of disease
Using a threshold of 0.50, Patient A is classified as healthy.
But what if the cost of missing a sick patient is much higher than the cost of sending a healthy patient for additional testing?
You might deliberately choose a threshold of 0.30 instead.
Now:
0.42 → Disease
0.73 → Disease
You've changed the model's behavior without retraining the model.
That's a crucial idea:
The model produces probabilities. The threshold determines how those probabilities become decisions.
Why this matters in production
Different applications have different costs for mistakes.
For example:
| Application | Potentially costly mistake |
|---|---|
| Fraud detection | Missing actual fraud |
| Cancer screening | Missing a disease |
| Spam filtering | Blocking a legitimate email |
| Loan approval | Incorrectly rejecting a customer |
| Intrusion detection | Missing an actual attack |
There is no universal rule saying that 0.5 is the correct threshold.
The appropriate threshold depends on the business or operational cost of false positives and false negatives.
Accuracy: Useful, But Easy to Fool
Accuracy is probably the first metric most developers reach for.
The formula is straightforward:
Accuracy = Correct Predictions / Total Predictions
In other words:
Accuracy = (TP + TN) / (TP + TN + FP + FN)
Where:
- TP = True Positives
- TN = True Negatives
- FP = False Positives
- FN = False Negatives
It sounds sensible.
But there's a major problem.
The 99% Accuracy Trap
Imagine a fraud-detection system processing 10,000 transactions.
Only 1% are fraudulent.
That means:
9,900 legitimate transactions
100 fraudulent transactions
Now imagine our model is completely useless.
It predicts:
"Not fraud" for every transaction.
How accurate is it?
It correctly identifies all 9,900 legitimate transactions.
So:
Accuracy = 9,900 / 10,000
= 99%
99% accuracy.
For a fraud detector that catches zero fraud, that's obviously not a useful model.
This is the classic class imbalance problem.
When one class vastly outnumbers the other, accuracy can hide poor performance on the minority class.
When should you use accuracy?
Accuracy can still be useful when:
- Classes are reasonably balanced.
- False positives and false negatives have similar consequences.
- You care about overall correctness.
But when your positive class is rare, don't evaluate the model using accuracy alone.
Fraud, disease detection, anomaly detection, and security problems are classic examples where you should look deeper.
Precision vs. Recall
Now we get to two of the most important metrics in classification:
Precision and Recall.
The easiest way to understand them is with a concrete example.
Let's build a spam filter.
Our model predicts whether an email is:
Spam
Not spam
There are four possible outcomes.
True Positive
The email is actually spam, and the model says spam.
Actual: Spam
Predicted: Spam
Good catch.
True Negative
The email isn't spam, and the model says it isn't spam.
Actual: Not spam
Predicted: Not spam
Also good.
False Positive
The email isn't spam, but the model says it is.
Actual: Not spam
Predicted: Spam
This is potentially painful.
Your model just threw a legitimate email into the spam folder.
False Negative
The email is spam, but the model says it isn't.
Actual: Spam
Predicted: Not spam
The spam gets through.
These four outcomes form the foundation of classification metrics.
Precision: "When I Predict Positive, How Often Am I Right?"
Precision focuses on the predictions your model labeled as positive.
Precision = TP / (TP + FP)
In plain English:
Of everything the model predicted as positive, how much was actually positive?
Suppose our spam filter flagged 100 emails as spam.
Of those:
90 were actually spam
10 were legitimate
Then:
Precision = 90 / (90 + 10)
= 90%
A precision of 90% means:
When the model says "spam," it's correct about 90% of the time.
Precision becomes especially important when false positives are expensive.
For example, imagine a fraud system automatically blocks transactions.
A false positive means:
"This legitimate transaction is fraud."
Too many false positives could frustrate legitimate customers.
Recall: "How Much of the Positive Class Did I Find?"
Recall asks a different question.
Recall = TP / (TP + FN)
In plain English:
Of all the actual positive cases, how many did the model successfully find?
Suppose there are 100 spam emails in our dataset.
Our model catches 90 of them.
It misses 10.
Then:
Recall = 90 / (90 + 10)
= 90%
So the model has a recall of 90%.
Recall is particularly important when false negatives are expensive.
Consider a disease-screening system.
A false negative means:
"The patient has the disease, but the model says they don't."
That could be far more consequential than sending some healthy patients for additional testing.
Precision and Recall: The Seesaw
Here's where classification gets interesting.
You generally can't maximize precision and recall simultaneously just by moving the threshold around.
Why?
Because changing the threshold changes which observations are classified as positive.
Imagine our model produces these probabilities:
0.12
0.21
0.34
0.46
0.58
0.67
0.79
0.91
With a threshold of 0.50, we classify four observations as positive:
0.58
0.67
0.79
0.91
Now lower the threshold to 0.30.
Suddenly we classify six observations as positive:
0.34
0.46
0.58
0.67
0.79
0.91
We've become more aggressive about predicting the positive class.
That can help us find more actual positives, increasing recall.
But it can also cause us to incorrectly label more negative examples as positive, reducing precision.
Conversely, raising the threshold generally makes the model more conservative:
Higher threshold
↓
Fewer positive predictions
↓
Usually fewer false positives
↓
Precision can increase
But...
↓
More actual positives may be missed
↓
Recall can decrease
This is the precision-recall trade-off.
The important takeaway isn't that one metric is "better."
It's that the right balance depends on the problem you're solving.
Calculating the Metrics with scikit-learn
Let's assume we've already trained our logistic regression model.
A clean evaluation workflow looks something like this:
from sklearn.metrics import (
accuracy_score,
precision_score,
recall_score,
roc_auc_score,
)
# Binary predictions using the default 0.5 threshold
y_pred = model.predict(X_test)
# Probability of the positive class
y_proba = model.predict_proba(X_test)[:, 1]
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred, zero_division=0)
recall = recall_score(y_test, y_pred, zero_division=0)
roc_auc = roc_auc_score(y_test, y_proba)
print(f"Accuracy: {accuracy:.3f}")
print(f"Precision: {precision:.3f}")
print(f"Recall: {recall:.3f}")
print(f"ROC-AUC: {roc_auc:.3f}")
There's an important detail here:
roc_auc_score(y_test, y_proba)
Notice that we're passing probabilities, not y_pred.
That's intentional.
ROC-AUC evaluates the model across possible classification thresholds, so we want the model's continuous probability scores rather than predictions produced using one particular threshold.
ROC Curve: Evaluating More Than One Threshold
If threshold selection is so important, how can we evaluate a model without committing to one threshold?
Enter the ROC curve.
ROC stands for:
Receiver Operating Characteristic
A ROC curve plots:
True Positive Rate
vs.
False Positive Rate
The true positive rate is another name for recall:
TPR = TP / (TP + FN)
The false positive rate is:
FPR = FP / (FP + TN)
So the ROC curve essentially asks:
"As I change the classification threshold, how does my ability to catch positives change relative to how many negatives I incorrectly flag?"
Imagine starting with a very high threshold:
threshold = 0.95
Only extremely confident predictions become positive.
You'll generally have:
- Few false positives
- Few true positives
Now lower the threshold:
threshold = 0.80
More observations become positive.
You'll potentially catch more actual positives, but you'll also flag more negative cases.
Continue lowering it:
0.80 → 0.70 → 0.60 → 0.50 → 0.40 → ...
Each threshold produces a different point on the ROC curve.
Plotting a ROC Curve in Python
With scikit-learn, plotting a ROC curve is straightforward.
import matplotlib.pyplot as plt
from sklearn.metrics import RocCurveDisplay
RocCurveDisplay.from_predictions(y_test, y_proba)
plt.title("ROC Curve")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.grid(alpha=0.3)
plt.show()
You can also calculate the underlying values yourself:
from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(y_test, y_proba)
plt.figure(figsize=(7, 5))
plt.plot(fpr, tpr, label="Logistic Regression")
plt.plot([0, 1], [0, 1], "--", label="Random classifier")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.title("ROC Curve")
plt.legend()
plt.grid(alpha=0.3)
plt.tight_layout()
plt.show()
The dashed diagonal represents a classifier with no discriminatory ability.
The closer a useful ROC curve gets toward the top-left corner, the more effectively the model separates the two classes.
ROC-AUC: One Number for the ROC Curve
Looking at an entire curve is useful, but sometimes you need a single summary statistic.
That's where ROC-AUC comes in.
AUC means:
Area Under the Curve
The score ranges from 0 to 1.
A rough interpretation is:
AUC ≈ 0.50
→ No better than random ranking
AUC > 0.50
→ Some ability to distinguish classes
AUC → 1.00
→ Very strong class separation
For example:
ROC-AUC = 0.50
means the model has essentially no ability to distinguish the classes based on its ranking.
A model with:
ROC-AUC = 0.90
has substantially better ability to rank positive examples above negative examples.
But there's an important nuance.
ROC-AUC is not the same thing as accuracy.
And it isn't the same thing as precision or recall.
A model can have a strong ROC-AUC while still requiring careful threshold selection for your actual application.
What Does AUC Actually Mean?
Here's a useful mental model.
Take one randomly selected positive example and one randomly selected negative example.
A ROC-AUC value can be interpreted as the probability that the model assigns a higher score to the positive example than to the negative example.
So if:
ROC-AUC = 0.80
you can think of that as roughly an 80% probability that the model will rank a randomly chosen positive case above a randomly chosen negative case.
This makes ROC-AUC fundamentally about ranking and discrimination, rather than one particular classification threshold.
Accuracy vs. Precision vs. Recall vs. ROC-AUC
At this point, it's useful to keep the metrics straight.
| Metric | Main question |
|---|---|
| Accuracy | How often is the model correct overall? |
| Precision | When the model predicts positive, how often is it right? |
| Recall | Of all actual positives, how many did it find? |
| ROC-AUC | How well does the model distinguish positives from negatives across thresholds? |
Each metric tells you something different.
That's why evaluating a classifier with a single number can be misleading.
A Practical Evaluation Workflow
When evaluating a binary logistic regression model, I recommend thinking in layers.
1. Start with the confusion matrix
Before interpreting metrics, understand the raw counts.
Predicted
Negative Positive
Actual
Negative TN FP
Positive FN TP
This immediately tells you what kinds of mistakes the model is making.
2. Calculate accuracy—but don't stop there
accuracy_score(y_test, y_pred)
Accuracy is useful context.
But if your classes are heavily imbalanced, treat it cautiously.
A 99% accuracy score could mean:
Excellent model
or:
Model predicts the majority class every time
The confusion matrix helps you distinguish between those situations.
3. Check precision and recall
precision_score(y_test, y_pred, zero_division=0)
recall_score(y_test, y_pred, zero_division=0)
Ask:
Is a false positive more problematic?
Pay closer attention to precision.
Ask:
Is a false negative more problematic?
Pay closer attention to recall.
In many real-world systems, you'll care about both.
4. Evaluate ROC-AUC using probabilities
roc_auc_score(y_test, y_proba)
This tells you how well the model separates the classes across thresholds.
It also avoids tying your evaluation to the arbitrary assumption that:
threshold = 0.50
5. Choose the threshold deliberately
Once you've established that the model has useful predictive power, choose a threshold based on the application's requirements.
For example, you might decide:
Threshold = 0.30
because missing positive cases is particularly costly.
Or:
Threshold = 0.80
because false alarms are expensive.
The threshold should be treated as a modeling decision, not an unquestioned default.
One More Important Detail: Don't Tune on the Test Set
There's a common evaluation mistake worth calling out.
Suppose you try 20 different thresholds on your test set and pick the threshold that gives you the best recall/precision combination.
You've now used the test set to make a modeling decision.
That's data leakage into your evaluation process.
A better workflow is:
Training data
↓
Train logistic regression
↓
Validation data
↓
Choose threshold
↓
Final test data
↓
Report final performance
If you're using cross-validation, threshold selection can be incorporated into the validation process as well.
The test set should remain your final, untouched estimate of generalization performance.
A Production-Friendly Evaluation Function
For repeated model evaluation, it's useful to wrap the metrics into a small function.
from sklearn.metrics import (
accuracy_score,
precision_score,
recall_score,
roc_auc_score,
)
def evaluate_binary_classifier(model, X_test, y_test, threshold=0.5):
"""Evaluate a binary classifier at a specified threshold."""
y_proba = model.predict_proba(X_test)[:, 1]
y_pred = (y_proba >= threshold).astype(int)
return {
"threshold": threshold,
"accuracy": accuracy_score(y_test, y_pred),
"precision": precision_score(
y_test,
y_pred,
zero_division=0,
),
"recall": recall_score(
y_test,
y_pred,
zero_division=0,
),
"roc_auc": roc_auc_score(y_test, y_proba),
}
metrics = evaluate_binary_classifier(
model,
X_test,
y_test,
threshold=0.5,
)
for metric, value in metrics.items():
print(f"{metric}: {value:.3f}")
The nice thing about this pattern is that you can evaluate different thresholds without retraining the model:
for threshold in [0.3, 0.5, 0.7, 0.9]:
metrics = evaluate_binary_classifier(
model,
X_test,
y_test,
threshold=threshold,
)
print(metrics)
That makes the precision/recall trade-off much easier to investigate.
The Big Picture
Evaluating a binary logistic regression model isn't about finding one magical metric.
It's about understanding what the model gets right, what it gets wrong, and how those errors behave as the decision threshold changes.
The mental model I'd keep is:
Logistic Regression
│
▼
Probability scores
│
▼
Choose threshold
│
▼
Binary predictions
│
┌──────────────┼──────────────┐
▼ ▼ ▼
Accuracy Precision Recall
│
▼
Evaluate all thresholds
│
▼
ROC-AUC
And remember the key lessons:
- Don't blindly trust the 0.5 threshold. It's a default, not a law.
- Accuracy can be dangerously misleading with imbalanced classes.
- Precision measures the quality of positive predictions.
- Recall measures how many actual positives you successfully find.
- False positives and false negatives have different real-world costs.
- Changing the threshold shifts the precision/recall trade-off.
- ROC-AUC evaluates class separation across thresholds rather than at just one threshold.
- Use probability scores when calculating ROC-AUC.
- Choose your production threshold using validation data and the actual cost of errors.
The most important shift in thinking is this:
Don't ask, "What's my model's accuracy?" Ask, "What kinds of mistakes does my model make, and are those mistakes acceptable for the problem I'm solving?"
That's the difference between simply training a classifier and actually evaluating one.
Top comments (0)