Half a day lost is the honest cost of treating an AI model like a document. I discovered that the hard way: a CAPA opened for a data-provenance gap rolled forward into missing documentation, which then exposed weaknesses in change control and supplier traceability. This is what happened, what we changed, and the small automation that stopped the loop from repeating.
The trigger: a CAPA that looked simple and wasn't
An engineer flagged a discrepancy between on-device inference behaviour and the validation test bench. The CAPA looked routine: reproduce, find root cause, correct datasets or model weights. Quickly it turned into:
- We couldn't identify which training dataset produced the deployed model (no manifest, only folder names).
- Preprocessing steps changed between runs (different label encodings, a silent resampling step).
- Model binaries were overwritten in a shared location without an immutable model registry entry.
- Change control only referenced a release ticket number — not the dataset or container image digest.
What began as a data-provenance finding became a documentation finding, then a change-control finding. Auditors would call this a traceability gap. The EU AI Act (and notified bodies increasingly expect traceability for high‑risk AI components) means you must show how a model version ties to the data, the training pipeline, the verification evidence, and the approval record. We didn't have that linkage.
By midday my filter coffee was cold and I had a long list of evidence to assemble.
Why CMOs see this differently
As a CMO handling components and supplier networks, our "models" are often supplier-provided (analytics, inspection classifiers, OCR of COAs), or built from datasets stitched from multiple vendors. The usual eQMS workflows assume a device maker controls the full pipeline. They rarely fit a supplier-heavy reality where:
- Sub-tier suppliers supply datasets or models.
- Incoming inspection depends on vendor-provided models for automated checks.
- Supplier COAs, not internal datasets, feed the training corpus.
So when traceability requirements say "link model to data, verification, and approval" the practical work is: link supplier COA → dataset manifest → model artifact → CAPA/change control. That chain must be live and reviewable.
What we changed — practical steps that closed the CAPA
We focused on two goals: make the mapping explicit, and make evidence machine-readable so automation can help.
- Capture immutable artifacts at training/deployment time
- Enforce a model registry entry per build: model name, semantic version, container image digest, git commit hash, training job ID.
- Store a dataset manifest with checksums (file-level SHA256), source supplier IDs, date received, and preprocessing steps.
- Link artifacts to quality records
- Change control and CAPA templates now require the model registry ID and dataset manifest hash as mandatory fields.
- Verification reports must include the model ID used in the test and the dataset manifest referenced.
- Tighten supplier evidence capture
- Incoming COAs and supplier data drop now write a record to our Postgres table with supplier ID, file digest, and an ingest ticket number our QA team uses downstream.
- Update SOPs and sign-off rules
- SOPs clarified who can approve a model change and when a human sign-off is mandatory (no auto-close of model changes).
- We added a CAPA-driven risk assessment step: minor data-label fixes may be documented, larger preprocessing changes escalate to change control.
- Small automation to keep the chain alive
- A Python script hashes new dataset drops, creates dataset manifests, and pushes registry entries into Postgres.
- Grafana dashboards show model → dataset → change control links; alerts fire if a deployed container digest lacks a dataset manifest.
These are low-tech, low-friction moves. The big win was mandatory linking: unless a model change referenced the dataset manifest and the change control record, it could not be promoted to production.
How this affects automated and AI-assisted CAPAs
We are sceptical of "auto-close" features for CAPAs and of black-box AI-guided fixes. Controlled assistance works:
- AI-assisted CAPA assistance can suggest likely root causes (e.g., label drift, preprocessing mismatch) based on artifact differences, but the engineer must approve and sign the CAPA content.
- Automated CAPAs (e.g., auto-created evidence requests) are useful for consistency — but a human must review any closure that affects patient‑facing or inspection automation.
This aligns with a simple, defensible posture: AI proposes, human approves. It's also technically auditable because we tied the AI outputs back to immutable artifacts (manifest hashes, container digests) and the approval trail.
What I automated (so I wouldn't do this again)
A half-day chasing links is not scalable. I wired these bits into an existing Python + Postgres + Grafana stack we already use for incoming inspection:
- An ingestion job creates dataset manifests and stores checksums.
- A webhook from our CI creates model-registry entries with container digest and git hash.
- A Grafana panel surfaces mismatches (deployed digest without manifest, change control without model ID).
- A lightweight checklist enforces human approval before closing CAPAs that touch models.
The result: CAPA closure now includes a minimal set of traceable artifacts. Auditors get a clear chain. Engineers spend time fixing, not scavenging.
Practical takeaways
- Treat models as first-class QMS artifacts: dataset manifest + model registry entry + verification report is the minimum.
- Make the linkage mandatory in change control and CAPA templates.
- Keep the human in the loop for approvals — AI proposes, human signs.
- Automate the boring, repeatable bits (checksums, manifests, dashboard alerts) so people can focus on root cause.
I mention qmsWrapper in our team discussions because it targets device makers' traceability needs; as a CMO with dozens of suppliers, the supplier-linkage problems still require custom automation and supplier‑specific ingest. A single-vendor eQMS rarely covers all the edge cases when your data and models come from many partners.
What traceability practice have you added to prevent an AI-related CAPA from spiralling into documentation or change‑control chaos?
I work on qmsWrapper and am sharing an honest read of where it does not fit our supplier-heavy CMO workflows.
Top comments (0)