Catching Input Drift Before Accuracy Falls: A PSI Walkthrough

A deployed model can get worse without anyone touching its code or weights. The world it sees changes: users behave differently, upstream systems change units or formats, a new product category appears, or a sensor gets replaced. This is often called skill drift, and it is one of the least visible ways a model fails.

This article covers one technique you can apply today: comparing the distribution of live inputs against a fixed reference window using the Population Stability Index (PSI). Then we'll show how to confirm what the numbers mean once labels arrive.

Why accuracy alone is not enough

The obvious monitor is accuracy, but it has a timing problem. To compute it you need ground-truth labels, and in many systems those arrive days or weeks late, or only for a sample. Input distributions usually shift first, so watching inputs and predictions gives you an earlier signal while you wait for labels.

Input monitoring does not replace performance monitoring. It tells you where to look.

Step 1: Freeze a reference window

Pick a period when the model was performing well and the data looked like what it was built for. Save that data as a snapshot and treat it as your baseline. Do not silently update it. If you re-baseline, do it deliberately and write down why, because a baseline that moves with the data can hide drift entirely.

Step 2: Choose what to watch

Watch three kinds of signal:

• Numeric input features, such as transaction amount or session length.
• Categorical input features, compared by category proportions.
• The model's output score or predicted class distribution.

The output distribution is often the most useful single series because it reflects every input at once.

Step 3: Measure shift with PSI

PSI compares two distributions by splitting them into bins and summing a term for each bin:

PSI = sum over bins of (a_i - e_i) * ln(a_i / e_i)

Here e_i is the share of reference data in bin i, and a_i is the share of current data in bin i. The bin edges come from the reference data, so the reference defines what "normal" looks like.

A common convention reads PSI below 0.1 as little change, 0.1 to 0.25 as moderate change, and above 0.25 as major change. Treat these as starting points rather than laws. Calibrate them against your own history, since acceptable variation depends on the feature and the domain.

Here is a dependency-light implementation using NumPy:

import numpy as np

def psi(reference, current, bins=10, eps=1e-6):
    reference = np.asarray(reference, dtype=float)
    current = np.asarray(current, dtype=float)
    edges = np.quantile(reference, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    edges = np.unique(edges)
    e_counts, _ = np.histogram(reference, bins=edges)
    a_counts, _ = np.histogram(current, bins=edges)
    e = e_counts / max(e_counts.sum(), 1) + eps
    a = a_counts / max(a_counts.sum(), 1) + eps
    return float(np.sum((a - e) * np.log(a / e)))

Sanity-check it before trusting it. Passing the reference against itself should return a value very close to zero. Passing a deliberately shifted sample, such as one with the mean moved by a standard deviation, should return a clearly larger value.

The eps term prevents division by zero and log of zero when a bin is empty. np.unique removes duplicate edges, which can happen when a feature has many tied values; that means you may get fewer than ten bins, and that is fine.

Step 4: Establish a noise floor

PSI on small samples is noisy. Before setting alerts, estimate how much the value wobbles when nothing has changed. Resample your reference data with replacement, compute PSI between each resample and the full reference, and look at the spread. If the typical value is 0.04 with a range that reaches 0.08 under no real change, an alert threshold of 0.1 is reasonable and an alert at 0.05 will fire constantly.

Also set a minimum sample size per run. Below a few hundred rows, bin proportions can swing enough to make the result meaningless. Skip the calculation or flag it as low-confidence rather than alerting on it.

Step 5: Run it on a schedule

Choose a cadence that matches your volume. High-traffic systems can compute PSI daily over the previous day's data. Lower-volume systems may need weekly windows to reach the minimum sample size. Store each result with its timestamp so you can plot trends, because a slow climb over months is easier to act on than a single spike.

Step 6: Confirm against outcomes

When labels arrive for the same window, compute your performance metric and compare it with the drift results. The combinations tell you different things:

Input drift	Performance change	Likely meaning	Next action
High	Drop	Real drift affecting the model	Investigate, then retrain or adjust
High	None	Shift the model handles well, or a benign change	Document it and keep watching
Low	Drop	Label shift, concept change, or a pipeline bug	Check labels and the scoring pipeline first
Low	None	Normal operation	No action

The third row is the one people miss. If inputs look stable but performance falls, suspect the labels, the target definition, or a bug in how predictions are joined to outcomes.

Step 7: Respond with a runbook

When an alert fires, work through a fixed order:

1. Check upstream changes first. New schema versions, unit changes, missing-value spikes, and new category values cause a large share of alerts and are usually quick to fix.
2. If the shift is real, confirm which features drive it. Compare per-feature PSI to find the largest contributors.
3. Retrain or fine-tune on recent data that includes the new conditions. Validate on a holdout drawn from the new period, not only on older data.
4. Re-baseline the reference window deliberately and record the reason in your change log.

Common mistakes

• Using the original training set as the only reference long after the environment has changed.
• Setting thresholds without a noise floor, which trains the team to ignore alerts.
• Treating drift as proof that accuracy has fallen. Drift is a reason to check, not a verdict.
• Monitoring only numeric features and forgetting missing-value rates and categorical proportions.
• Updating the baseline automatically, which erases the evidence you need to see slow drift.

Wrap-up

PSI against a frozen reference gives you an early, label-free signal that inputs have moved. Pair it with a noise floor so alerts mean something, and with delayed performance metrics so you know whether the shift matters. The method is small enough to add to an existing pipeline in an afternoon.

If you want a longer, step-by-step version with templates and checklists, the full Identify and Prevent AI Skill Drift guide is available at https://heracles-store.pages.dev/p/identify-and-prevent-ai-skill-drift/.

---

Written with AI assistance and reviewed for accuracy before publishing.