Healthcare AI Needs to Account for Practice Drift

Machine learning systems are usually evaluated against a relatively stable development environment. Healthcare rarely provides one.

Clinical practice changes continuously through guideline updates, new treatments, diagnostic technologies, workflow redesign, staffing changes, and documentation practices.

These changes can modify the data-generating process.

Consider a model trained to predict a clinical outcome using historical patient information. If treatment protocols later change, the relationship between the same patient characteristics and the outcome may change. The model may therefore experience degraded clinical validity even when conventional input distributions do not show dramatic drift.

This is practice drift.

It differs from ordinary feature drift because the underlying clinical process has changed.

A useful monitoring strategy should therefore combine model performance monitoring with clinical governance signals. Significant changes in guidelines, care pathways, diagnostic technologies, treatment patterns, or documentation should trigger review.

For some systems, the appropriate response may be recalibration. Others may require retraining or external validation. High-consequence systems may need temporary restrictions while their assumptions are reassessed.

Agentic healthcare AI makes this especially important. An agent can execute decisions consistently according to its objective, but consistency does not guarantee that its objective remains aligned with current clinical practice.

The technical lesson is simple: monitor the environment that generates the data, not only the data itself.