Traditional machine learning evaluation often assumes that the relationship between inputs and outcomes is relatively stable.
Deployment can challenge that assumption.
Consider a clinical risk model that identifies patients for additional monitoring.
Before deployment, monitoring decisions are made without the model.
After deployment, clinicians use the model to select patients.
The intervention changes.
The observations change.
The outcomes may change.
The resulting data is generated under a different decision process.
This creates a feedback loop between the model and the environment.
Several mechanisms can produce this effect.
A model can influence which patients receive tests.
It can influence treatment decisions.
It can change referral patterns.
It can change documentation.
It can alter resource allocation.
It can change the timing of interventions.
These changes can affect both future outcomes and the availability of future labels.
This has important implications for healthcare AI evaluation.
A model that performs well on historical data may behave differently when clinicians begin acting on its predictions.
Repeated evaluation is therefore important.
Teams may need to monitor changes in feature distributions, intervention patterns, outcome rates, alert responses, subgroup performance, and calibration after deployment.
The data-generating process itself may need to be documented.
For agentic systems, the feedback problem can become even more complex because the system may directly execute actions rather than merely provide recommendations.
The agent can change the workflow that produces the next generation of data.
This does not make deployment impossible.
It makes system-level evaluation necessary.
Healthcare AI should be treated as an intervention in a socio-technical system.
The model is not merely predicting an environment.
Once deployed, it can help shape that environment.
Top comments (0)