If your SaMD calls a foundation model you don't pin, you can't explain patient-specific outputs

I started worrying about this during a product-risk review: our clinical decision-support prototype relied on a hosted foundation model for part of its inference chain. We validated behavior end-to-end, wrote test vectors, and documented performance. Then the provider rolled a silent model update and one of our regression prompts changed behavior enough that our acceptance checks would have failed for certain edge cases.

That scenario is the exact thing that breaks reproducibility for SaMD. If a regulator, notified body, or physician asks "why did the AI say that for patient X on date Y?", you need an answer rooted in auditable artifacts — model version, input, runtime, and provenance. If the underlying model silently version-bumps you, that answer is gone.

Why reproducibility matters for SaMD (short list)

• Patient safety depends on repeatable behavior. If an output changes without a controlled change, risk assessments and mitigations can be invalidated.
• FDA expectations (AI/ML SaMD guidance) and IMDRF work on SaMD emphasize lifecycle management and traceability for software that impacts clinical outcomes.
• The EU AI Act (Article 12 and surrounding transparency obligations) will require traceable and explainable outputs for high-risk AI — you can't meet that if you cannot reproduce the inference environment.

The silent-version-bump problem

Foundation models deployed as managed services introduce a dependency you don't control. Two practical failure modes we worry about:

• A provider replaces or fine-tunes a base model and the same API name now maps to a different binary/weights — you receive different outputs for identical inputs.
• The provider changes tokenization, pre/post-processing, or default temperature/decoding settings, creating subtle drift.

Either way, your audit trail lacks the unique identifier necessary to tie an inference back to a specific, immutable artifact.

What I treat as non-negotiable controls (what we've started doing)

These are practical controls that I push for in our QMS and validation plans for any SaMD that uses external models.

• Pin the model or require an immutable artifact
  • If you can, use a model registry or host an artifact you control (MLflow, DVC, or an OCI image with model binary). If you must use a vendor-hosted model, contractually require immutable model versioning or a "model digest" you can reference.
• Record exhaustive inference metadata (for every call)
  • timestamp, patient/input ID (de-identified if needed), exact input payload, prompt template, model provider, model_id, model_version/model_digest, runtime env, randomness seed, model outputs, and confidence/score.
  • Store logs in an append-only store and tie them to your traceability records.
• Treat model changes as controlled design changes
  • Any model update (provider or internal) must go through change control: risk assessment, performance re-validation on locked test sets, clinical impact review, and release notes that are versioned in the QMS.
• Keep offline reproducibility artifacts
  • Snapshot the code, container images, model weights (or a certified hash), and test datasets used for validation in your configuration management system.
• Shadow and staged deployment
  • Run new model versions in shadow mode against production inputs and compare outputs before any switch. Define acceptance gates and rollback criteria.
• Vendor obligations in contract
  • Require timely notification of model updates, release notes, and — ideally — the ability to pin to a specific model digest or roll back. Insist on retention of historical model artifacts for audit windows.

Operational tips for audits and investigations

• Build an "explainability bundle" for each inference that contains:
  • the inference metadata (above),
  • the relevant model card or vendor release notes,
  • the snapshot of the validation dataset used for acceptance of that model version.
• Log provenance at the system boundary
  • It's fine to use a managed service, but your system must record the provider's response headers (model id/version) and persist them.
• Avoid ephemeral randomness in production
  • Where determinism is required for traceability, use fixed seeds or decoding strategies that produce reproducible outputs (and document this choice).

Tradeoffs and real-world constraints

• Pinning models and storing weights increases operational and security burden (storage, IP constraints, licensing). Sometimes the vendor simply won't allow artifact export.
• Shadow testing and robust change control add latency to deploying beneficial updates. You have to balance clinical benefit from model improvements with the regulatory need for controlled change.
• Explainability methods are imperfect. Model cards and local explainers help, but they don't replace a reproducible artifact and a logged inference trail for forensic questions.

Where QMS and ML ops meet

From a QMS perspective, model updates are software changes. They require design history file entries, change-impact assessments, testing records, and traceability to clinical risk. From an MLOps perspective, you need model registries, immutable artifacts, and reproducible inference logging. The two must be connected: your QMS needs to ingest model registry metadata as configuration items so auditors can follow the chain.

If you're building or validating SaMD today, assume regulators will ask for provenance on individual patient-impacting outputs. Plan to answer "why?" with data, not post-hoc rationalization.

How are other SaMD teams handling foundation model dependencies in practice — vendor locks with contractual model pinning, or is self-hosting the only workable regulatory path?