Every clinical AI vendor shows you validation results. Almost none of them were produced on your population. This post is about the infrastructure that closes that gap, and why it has to be yours rather than theirs.
Why vendor validation does not transfer
Model performance in healthcare is unusually sensitive to three things that vary enormously between institutions.
Case mix. Prevalence changes the positive predictive value of a fixed-sensitivity model directly. A sepsis model validated in a tertiary referral centre encounters a different base rate at a community hospital, and the same threshold produces a different alert burden.
Documentation practice. Models trained on free text or structured fields inherit the conventions of the site that produced the training data. Different templates, different macro usage, different specialty conventions — all shift the input distribution in ways that are invisible until measured.
Coding behaviour. Local code sets, mapping decisions, and the ten percent that did not map cleanly and got assigned something adjacent. This is the least discussed and frequently the largest source of divergence.
None of these are vendor failures. They are why local evaluation is not optional.
The components
1. A held-out local sample with defined ground truth
The hard part is ground truth, not sampling. For some tasks a downstream coded outcome is adequate. For most, you need adjudicated labels — clinicians reviewing a sample against a written rubric, with a second reviewer on a subset to measure agreement.
Budget for this honestly. A few hundred adjudicated cases is usually enough for a first read and is typically the most expensive line in the whole exercise. It is also the line most likely to be cut, which is how organisations end up evaluating against a proxy outcome that answers a different question.
2. Stratified performance reporting
Report performance overall and separately for the subgroups where failure would be most consequential. At minimum: age bands, sex, race and ethnicity as recorded, primary language, payer class, and site of care.
This is not a compliance exercise. Aggregate performance can be entirely acceptable while a subgroup with a different base rate or different documentation patterns is being systematically failed, and the aggregate number will never show it.
3. Operating point selection with the alert burden attached
A model is deployed at a threshold, not at an area-under-curve. Pick the operating point against the burden it produces — alerts per clinician per shift — and involve the people who will receive those alerts in choosing it.
Alert fatigue is the dominant failure mode for clinical decision support, and it is a threshold decision made by someone who never sees the consequence.
4. Scheduled re-evaluation
Model drift in healthcare is real and its failure mode is silent. Performance degrades gradually while clinician trust remains at the original level, because nothing is designed to notice.
Schedule the re-run — quarterly is a reasonable default — on a fresh local sample, with the same stratification. Define in advance what degradation triggers a review and who has the authority to withdraw the tool. If nobody has that authority, the monitoring is decorative.
5. Data plumbing
Practically, this means a reproducible extract, a stable identifier strategy that survives de-identification, and a mapping layer that reconciles local codes to standard terminologies with the unmapped remainder made explicit rather than silently dropped. Bulk FHIR export handles a lot of this now, with the caveat that any extract without a defined retention policy, access model and de-identification standard is a liability that grows with every run.
What to demand at contract time
Put these in the agreement, because retrofitting them is much harder:
- The right to evaluate on your own data before full deployment
- Access to model outputs at a granularity that supports subgroup analysis
- Notification when the vendor updates the model, with a defined re-evaluation window
- A documented statement on whether your inputs are retained or used for training
- An exit path that returns your data and does not strand the workflow
The fourth is a business associate agreement question and is frequently answered vaguely. Push for specificity.
Longer piece covering data readiness, interoperability, governance and engagement models: Healthcare IT Consulting Services: What Actually Moves the Needle in 2026. General ML engineering context in our machine learning development services guide.
Frequently Asked Questions
How many labelled cases do we need?
For a first read on a binary classification task, a few hundred adjudicated cases usually suffices, with more needed for rare outcomes or for meaningful subgroup precision. Power the subgroup analysis deliberately rather than hoping the sample covers it.
Can we use retrospective data or does it need to be prospective?
Retrospective is fine for initial evaluation and much cheaper. Prospective silent-mode running — the model produces predictions nobody acts on — is the stronger design for the deployment decision, because it exercises the real data pipeline rather than a curated extract.
Who adjudicates the ground truth?
Clinicians in the relevant specialty, working from a written rubric, with a second reviewer on a subset so you can report inter-rater agreement. Without that agreement number, a disappointing model result cannot be distinguished from noisy labels.
What drift threshold should trigger a review?
Define it as a relative degradation from your local baseline rather than an absolute value, set it before deployment, and set it separately for the subgroups you care about. The specific number matters less than agreeing it in advance and naming who acts on it.
Does this apply to ambient documentation tools too?
Yes, with different measures. Note quality against a rubric, blinded review, and error categories that matter clinically — fabricated content, omitted findings, incorrect attribution — rather than a single accuracy score.
Is bulk FHIR export enough to build this?
It covers a lot of the extraction, but not terminology reconciliation, ground truth, or governance. Treat it as the plumbing rather than the solution, and do not build an extract pipeline before deciding its retention and access policy.
Top comments (0)