Model Monitoring in Production: Catching Drift Before It Hits the Bottom Line

#mlops #machinelearning #monitoring #insurtech

Most teams deploy an ML model the way you'd install a boiler: commission it, confirm it works, walk away. Then months later the numbers drift, someone asks whether "the model" is to blame, and the honest answer is nobody knows — because nobody was watching it. An unmonitored model isn't a stable asset; it's a slowly rotting one, and the rot is invisible until it shows up in your metrics.

Two kinds of drift, watched separately

Data drift: the inputs change. The distribution of ages, locations, transaction types flowing in no longer matches training.
Concept drift: the relationship changes. Same inputs, different outcome. This is the dangerous one — inputs can look perfectly normal while the model's learned assumptions quietly stop holding.

Why the usual playbook fails in insurance

In many ML apps you catch a bad model fast because ground truth arrives quickly — the user clicked or didn't. Insurance is the opposite: the ground truth for a pricing decision is whether the policy was profitable, and you might not know for years. By the time labels prove the model drifted, you've written years of business on bad prices. You cannot wait for outcomes. You monitor the leading indicators — inputs and predictions — because outcomes arrive too late to save you.

What to actually monitor

Input distributions. Track each feature's live distribution vs the training baseline. A population stability index (PSI) breach is your earliest warning.

def psi(expected, actual, bins=10):
    import numpy as np
    e = np.histogram(expected, bins=bins)[0] / len(expected) + 1e-6
    a = np.histogram(actual,   bins=bins)[0] / len(actual)   + 1e-6
    return np.sum((a - e) * np.log(a / e))   # >0.2 => investigate

Prediction distributions. Suddenly scoring 15% more into the top risk band? Could be real — or a broken upstream feed. Either way, know today.
Data quality at the door. Null rates, out-of-range values, a categorical field that sprouted a new value. Most "model problems" are a pipeline feeding the model garbage.
Outcomes, when they arrive. Actual-vs-expected by segment — the slow, authoritative signal that confirms what the leading indicators warned about.

Monitoring ≠ observability

Data observability tells you the pipeline is healthy (job ran, table landed). Model monitoring tells you the model is still valid (the learned relationships hold). You need both; a pipeline can be perfectly healthy while it faithfully delivers the model into irrelevance.