Why accuracy alone can fool you on imbalanced datasets. Somewhere, an ML model is proudly reporting 99.4% accuracy.
The dashboard is green. The stakeholders are smiling. Someone is probably preparing the report.
Then a dangerous question appears:
“How much fraud did the model actually catch?”
The answer: zero 😟
Welcome to the accuracy trap.
Watch the full video:
Meet the world’s laziest model
Consider a simulated dataset containing 20,000 card transactions, where only 0.6% are fraudulent.
Now introduce our highly sophisticated baseline:
def lazy_model(transaction):
return "not fraud"
No training. No feature engineering. No hyperparameter tuning. No GPU trying to heat the neighbourhood.
It simply predicts “not fraud” every time.
And because almost every transaction is legitimate, the model achieves:
| Metric | Lazy Model |
|---|---|
| Accuracy | 99.4% |
| Precision | 0% |
| Recall | 0% |
| F1 score | 0% |
The model is correct most of the time but useful none of the time. It catches no fraud and probably still asks for a promotion.
Considering Precision and recall
Accuracy asks:
“How often was the model correct overall?”
That sounds reasonable until one class heavily outnumbers the other.
For fraud detection, two other metrics are far more revealing:
Precision asks:
“Of everything flagged as fraud, how much was actually fraud?”
Low precision means your system keeps blocking genuine customers. Congratulations—you have successfully detected someone buying groceries.
Recall asks:
“Of all the fraud that really happened, how much did we catch?”
Low recall means the fraudsters leave with the money while the model celebrates its excellent accuracy.
What happens with a real model?
We trained a logistic regression model using balanced class weights.
Its results looked less impressive at first:
| Metric | Lazy Model | Logistic Regression |
|---|---|---|
| Accuracy | 99.4% | 85.3% |
| Precision | 0% | 3.1% |
| Recall | 0% | 77.8% |
| F1 score | 0% | 6.0% |
The real model has lower accuracy, but it catches almost 78% of the fraud.
So which model is better?
The Lazy Model wins the dashboard beauty contest.
The logistic regression model wins the actual fraud-detection contest.
Precision and recall are professional rivals
There is usually a trade-off:
- Lower the fraud threshold and recall increases, but so do false alarms.
- Raise the threshold and precision may improve, but more fraud can slip through.
The “best” threshold is therefore not only a mathematical choice. It depends on business cost.
What is worse?
- Annoying a legitimate customer?
- Missing a fraudulent transaction?
- Sending 10,000 alerts to a fraud team with three analysts and one coffee machine?
The answer depends on the system.
The takeaway
For imbalanced classification problems, accuracy is not useless but it is often incomplete.
Always look at:
- Precision for alert quality
- Recall for detection coverage
- F1 score for balance
- The confusion matrix for the types of mistakes
- Business impact for what those mistakes actually cost
A 99.4% accurate model can still be terrible.
Metrics do not lie but they are perfectly happy to let us misunderstand them.
Top comments (1)
The cost-matrix framing at the end is the underrated part. Precision and recall are accounting identities for the same trade-off; what actually sets the threshold is what each error type costs your specific system. In RAG and LLM evaluation the trap recurs in a different form: a model that refuses or hedges every uncertain query can achieve near-zero hallucination rate while being functionally useless — high precision on what it does say, but recall on genuinely needed answers collapses. The right metric is always derived from the decision the system is supposed to make, not from the convenience of the label distribution.