You deployed a model three months ago. It was accurate then. It may not be now. Data distributions shift, label patterns change, and your model silently degrades — until someone notices the product is broken. The problem is not writing ML models; it's knowing when they stop working.
Why Models Drift
A model learns the statistical relationship between inputs and outputs at training time. When the world changes — customer behavior shifts, a new product launches, a data pipeline upstream changes format — the model keeps predicting as if nothing happened. There are two distinct failure modes:
Data drift: the input distribution P(X) has changed. Features that were rare are now common; ranges shifted.
Concept drift: the relationship P(Y|X) has changed. The same input now maps to a different output than the model expects.
Both are silent killers. Without monitoring, you find out from an angry customer, not a dashboard.
Detecting Data Drift with PSI and KS Tests
The Population Stability Index (PSI) is a simple, effective metric for detecting distribution shift in a single feature. It compares the distribution of that feature between a reference window (training or recent baseline) and the current production window.
import numpy as np
from scipy import stats
def compute_psi(reference: np.ndarray, current: np.ndarray, buckets: int = 10) -> float:
"""Compute PSI between two distributions."""
breakpoints = np.percentile(reference, np.linspace(0, 100, buckets + 1))
breakpoints[0] = -np.inf
breakpoints[-1] = np.inf
ref_counts, _ = np.histogram(reference, bins=breakpoints)
cur_counts, _ = np.histogram(current, bins=breakpoints)
ref_pct = ref_counts / len(reference)
cur_pct = cur_counts / len(current)
# Clip to avoid log(0)
ref_pct = np.clip(ref_pct, 1e-10, None)
cur_pct = np.clip(cur_pct, 1e-10, None)
return np.sum((cur_pct - ref_pct) * np.log(cur_pct / ref_pct))
def interpret_psi(psi: float) -> str:
if psi < 0.1:
return "stable"
elif psi < 0.2:
return "moderate_drift"
return "significant_drift"
PSI below 0.1 means the distribution is stable. Above 0.2 means something significant has changed and deserves investigation.
For numerical features, the Kolmogorov-Smirnov test is a good complement: it is non-parametric and does not require binning.
def ks_drift_test(reference: np.ndarray, current: np.ndarray, alpha: float = 0.05) -> dict:
statistic, p_value = stats.ks_2samp(reference, current)
return {
"statistic": round(statistic, 4),
"p_value": round(p_value, 4),
"drift_detected": p_value < alpha,
}
A p-value below 0.05 means you can reject the null hypothesis that both samples come from the same distribution. Use KS alongside PSI — PSI tells you the magnitude, KS tells you whether the difference is statistically significant given your sample size.
Building a Monitoring Pipeline
A monitoring system needs three components: a data store for predictions and inputs, a scheduler that runs checks on a fixed window, and an alerting layer that surfaces issues.
Here is a minimal architecture using SQLite for storage:
import sqlite3
import json
from dataclasses import dataclass
from datetime import datetime, timedelta
from typing import Optional
@dataclass
class PredictionRecord:
timestamp: datetime
features: dict
prediction: float
label: Optional[float] = None
def log_prediction(conn: sqlite3.Connection, record: PredictionRecord) -> None:
conn.execute(
"INSERT INTO predictions (ts, features_json, prediction, label) VALUES (?, ?, ?, ?)",
(
record.timestamp.isoformat(),
json.dumps(record.features),
record.prediction,
record.label,
),
)
conn.commit()
def fetch_window(conn: sqlite3.Connection, start: datetime, end: datetime) -> list[dict]:
cur = conn.execute(
"SELECT features_json, prediction, label FROM predictions WHERE ts BETWEEN ? AND ?",
(start.isoformat(), end.isoformat()),
)
return [
{"features": json.loads(row[0]), "prediction": row[1], "label": row[2]}
for row in cur.fetchall()
]
def run_drift_check(
conn: sqlite3.Connection,
feature_name: str,
reference_days: int = 30,
window_hours: int = 24,
) -> dict:
now = datetime.utcnow()
reference_start = now - timedelta(days=reference_days)
reference_end = now - timedelta(days=1)
current_start = now - timedelta(hours=window_hours)
reference_rows = fetch_window(conn, reference_start, reference_end)
current_rows = fetch_window(conn, current_start, now)
if not reference_rows or not current_rows:
return {"error": "insufficient_data"}
ref_values = np.array([r["features"][feature_name] for r in reference_rows])
cur_values = np.array([r["features"][feature_name] for r in current_rows])
psi = compute_psi(ref_values, cur_values)
ks = ks_drift_test(ref_values, cur_values)
return {
"feature": feature_name,
"psi": round(psi, 4),
"psi_status": interpret_psi(psi),
"ks_drift": ks["drift_detected"],
"checked_at": now.isoformat(),
}
This keeps monitoring logic close to the data. For larger volumes, the same pattern works with PostgreSQL or a time-series store like TimescaleDB — just swap the connection and query dialect.
Handling Concept Drift When Labels Arrive Late
The hardest case is concept drift when labels arrive with a delay — fraud detection, churn prediction, credit scoring. You cannot compare P(Y|X) in real time because the ground truth is not known yet.
Two practical workarounds:
Monitor proxy signals. If predictions are skewed toward one class by more than two standard deviations from the historical mean, something changed even without ground truth.
def prediction_shift_alert(
predictions: list[float],
reference_mean: float,
reference_std: float,
threshold: float = 2.0,
) -> bool:
current_mean = np.mean(predictions)
z_score = abs(current_mean - reference_mean) / reference_std
return z_score > threshold
Sample and label fast. Take a 1–5% random sample of predictions, have humans or a heuristic system label them quickly, and run accuracy checks on that subset. It is impractical at training scale but very practical for catching regressions early.
Putting It Into Production
A monitoring system that runs once a week is not monitoring. Schedule drift checks to run at least daily, and hourly for high-stakes models. Three principles that matter in practice:
Alert with context. Do not just send "drift detected" — include which feature, the PSI value, the reference window size, and the current window size. People learn to ignore vague alerts quickly.
Version your baselines. When you retrain and redeploy a model, update the reference distribution. Otherwise PSI will always be high after a redeployment and your on-call team will start ignoring it within a week.
Separate the monitoring writes from production writes. The monitoring service should read from your production database but write to a separate store. A monitoring bug should not cause a production incident.
For teams running AI systems, applying the same rigor to model quality tracking that you apply to infrastructure security hardening will surface degradation before it reaches your users.
The Takeaway
ML drift detection is not a machine learning problem — it is a software engineering problem. The statistics are straightforward: PSI, KS test, mean shift. The hard part is the pipeline: reliable scheduling, correct data retention, low-noise alerting routed to the right person.
Start small. Log every prediction with its input features. Add PSI checks for your ten most important features. Run a daily job that alerts if anything crosses 0.2. That baseline will catch the majority of drift incidents that would otherwise surface as customer complaints.
I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.
Top comments (0)