Last spring I sat in a risk management review where someone asked the obvious thing. If our SaMD retrained last Tuesday — quietly, on a new cohort — what in our risk file caught that?
There was a long silence. Then someone said, "well, the post-market surveillance plan covers drift."
It doesn't. Not the way we wrote it. And I think that's true for most Class II shops I'm aware of.
Here's the dry reminder I'm forcing on myself this quarter: a heft of policy documents doesn't stop a retrained model from reshaping risk profiles overnight. The risk file is a snapshot. The model is moving. If your QMS treats them as the same thing, you're going to get caught flat-footed at your next audit, or worse, at your next CAPA.
The gap I'm watching
Risk management under ISO 14971 is supposed to be living. In our shop it isn't. It's a controlled document that gets revisited at design reviews, at CAPA triggers, and at the annual management review. That's three or four scheduled touches against a system that may shift continuously.
For software as a medical device with a learning component, the underlying behavior can change without a single ECO. The model weights are different. The cohort distribution is different. The error profile on subgroups you didn't even know were subgroups — different. None of those changes walked through change control, because none of them were design changes in the traditional sense. They were model updates, drift corrections, retraining events that fell under our "algorithm maintenance" carve-out.
That carve-out was a gap. We knew it was a gap. We documented it as a known gap and moved on. Then the notified body picked it up.
What the standards actually ask
ISO 14971:2019 expects you to keep the risk management file current. IEC 62304 expects you to revisit software risk when the software changes. The EU AI Act, where it applies, layers in additional obligations around dataset shift, post-market monitoring, and serious incident reporting for high-risk AI systems.
The FDA's Predetermined Change Control Plan guidance is the closest thing to a recipe. You commit to a protocol in advance: what inputs change, what triggers a re-evaluation, what bounds you're operating in. If you stay inside the protocol, you don't file a new submission. If you don't have the protocol, every retraining event is potentially a new submission. Or a new audit finding.
The standards aren't saying anything new. They're saying: link the change to the risk. Continuously, if you can.
What documentation can't catch
A 47-page risk management procedure is great for onboarding. It's terrible for detecting Tuesday's retrain. The things that miss us:
- Drift in subgroup performance that doesn't trip a global alarm
- A new failure mode introduced by a feature the model learned from post-deployment data
- A labeling convention that quietly changed upstream
- A dependency update in the inference stack that altered numerical precision
None of those generate a controlled document event by default. They generate log entries. Your QMS is reading PDFs; the risk is in the JSON.
What I'm trying instead
Three things, in roughly this order, with mixed results:
- Trigger the risk file from telemetry, not from meetings. We wired a monitoring job to flag when subgroup accuracy moved by more than an agreed threshold. The flag opens a risk-management-workflow item, not an email thread. Whether the item turns into a CAPA, a labeling change, or a non-event is a documented decision.
- Write a real PCCP and stick to it. This is the part that hurts. You commit to a protocol that names the retraining triggers, the validation gates, and the rollback conditions. If you can't commit to that, you probably shouldn't be retraining in production.
- Make post-market data flow into design input. Not into a quarterly report. Into design input. The PMS plan should say, in writing, which signals feed back into which risk control. If your PMS deliverable is a memo, the loop is broken.
None of this is novel. None of it requires us to buy anything. It requires us to admit that the document control system is not the risk management system — it's a record of decisions. The risk management system is the system that detects, decides, and documents. The detection part is what most of us skip.
The honest part
I'll tell you what I'm still stuck on: who owns the trigger. QA doesn't run the monitoring system. Data science does. When the alert fires, it lands in their queue. The QMS workflow opens in ours. The handoff is where retraining-induced change slips through.
So here's what I want to ask anyone reading: where does the retrain event hand off to your QMS in your shop, and what makes that handoff actually happen — not theoretically, in the procedure, but on a Wednesday at 4pm?
Top comments (0)