In 2021, a team at Michigan Medicine ran an external validation of the Epic Sepsis Model across 38,455 hospitalizations. Epic's own documentation put the model's AUC somewhere between 0.76 and 0.83. Michigan measured 0.63. Sensitivity was 33%, meaning it missed about two-thirds of septic patients, and it still managed to fire alerts on 18% of everyone admitted.
The detail that should bother you as an engineer is in the inputs. One of the model's features was whether a clinician had ordered antibiotics. Ordering antibiotics means somebody has already suspected an infection. So the model was, in part, learning to predict a decision that had already been made, then congratulating itself on the prediction.
That is textbook leakage. It ran in production at hundreds of hospitals for years.
Why do healthcare AI pilots fail at scale?
Nobody caught it because the environment had no mechanism for catching it. There was no external validation until researchers did it on their own initiative, no post-deployment monitoring worth the name, and no owner whose job it was to ask whether the thing was still working.
This is the pattern underneath most stalled healthcare AI. The model is rarely the constraint. The real constraint is everything it plugs into: patient records spread across EHR instances that different departments bought in different decades, imaging sitting in a system that speaks to nothing, duplicate MRNs, coding standards that drifted three reorganizations ago, and operational data (scheduling, staffing, supply) that has never been joined to clinical data in any usable way.
Train on a curated dataset, and you get a demo. Point the same weights at live ingestion, and performance degrades quietly, which is the worst way for it to degrade.
What AI-ready infrastructure means
Four things, and they're all boring:
Interoperability. Data can actually move between clinical, operational, and administrative systems without a human exporting a CSV.
Governance: Someone owns data quality, someone owns access, while someone's name is on the model's performance next quarter.
Workflow integration: Output lands where the clinician already works. If it creates a parallel process, it creates one that gets eventually ignored.
A scalable operating model: Monitoring, retraining, and drift detection exist before you need them, not after a JAMA paper.
Miss one and you get pilot purgatory. Organizations that assess AI-ready infrastructure before procurement end up making very different buying decisions than those who discover the gaps at integration time, which is roughly when the budget is already committed.
Aged care shows the failure mode at its bluntest. Fall detection works fine in a lab. Deploy it into a facility where half the documentation is still paper, and the notification pathway was never wired into anyone's actual shift; also, the alert fires into a void. The system produces the appearance of intervention but the resident is still on the floor.
Questions worth asking before the demo:
- Can you access the same patient consistently across systems, or are you resolving identity by hand?
- Who owns data quality, by name?
- Have the alert escalation paths been tested with real people on shift?
- Is there a monitoring framework, or are you planning to find out from a study?
The unglamorous work is the real work. Nobody puts an interoperability audit in a board deck. It's still the difference between a system a nurse trusts at an unexpected time and one she's learned to click through.
Top comments (0)