Here's a model that predicts rockfalls with 99% accuracy. It's also completely worthless. If a dangerous slope fails, say, one day in a hundred, a model that shrugs and says "no rockfall" every single day is right 99% of the time — and will get people killed. That paradox is the first thing you confront when you build ML for rare, catastrophic events, and it shaped how I approached the rockfall prediction system.
Accuracy is a lie when one class is rare
Accuracy measures how often you're right overall. That sounds fine until the thing you care about almost never happens. When 99% of days are safe, "always predict safe" scores 99% while detecting exactly zero of the events that matter. The metric rewards the model for ignoring the entire reason it exists.
So for rare-event prediction, accuracy isn't just unhelpful — it's actively misleading. It hides catastrophic blindness behind a great-looking number. The question is never "how often is it right?" It's "of the real events, how many did it catch, and how many false alarms did that cost?"
Optimize for the error that actually hurts
Not all mistakes are equal here, and pretending they are is the core error. There are two ways to be wrong, and they have wildly different price tags:
- A false negative — missing a real rockfall — is the failure you cannot accept. Someone relied on the all-clear. This is the error you drive toward zero, even at a cost.
- A false positive — an alarm when nothing happens — is annoying and erodes trust, but nobody gets hurt. You want few of them, but you'll trade a handful of them to never miss the real thing.
That asymmetry is the whole design. You measure with recall (of the real events, how many did we catch) and AUC-ROC (how well the model separates danger from safety across every threshold) instead of accuracy, and you tune the decision threshold toward catching events, accepting more false alarms as the price of never missing the one that counts.
The takeaway
The hardest part of rare-event ML isn't the model — it's refusing to be seduced by a high accuracy score. The moment your positive class is rare and expensive to miss, accuracy becomes a comfortable lie, and the real work is choosing metrics and thresholds that reflect what a mistake actually costs in the world.
Building the rockfall system taught me to design around the cost of being wrong, not the frequency of being right. The full pipeline — sensor fusion, the model, the alerting — is on the project page.
👉 See it: www.divyakush.com/projects/rockfall-prediction
Divyakush Punjabi — Full-Stack & AI Systems Engineer
🌐 https://www.divyakush.com · 💼 LinkedIn · 💻 GitHub
Top comments (0)