A Synthetic-First Quality Assurance Pipeline for Pre-Deployment Clinical AI Validation

Moumita Ghosh — Sat, 14 Nov 2026 14:40:35 +0000

Testing clinical AI models presents a paradox: rigorous validation needs realistic data, but using real patient data for testing violates HIPAA. Generic MLOps tools don't close this gap because they carry no clinical context. This write-up describes Qualisera AI, a pre-deployment quality assurance framework that validates healthcare AI workflows using 100% synthetic and public test data. Through declarative clinical safety gates and automated API testing, the current build executes 113 automated checks in a single run — 100 dataset-focused scenarios (51 positive, 49 negative) plus 13 supporting tests for the API, dashboard, and loaders — with zero failures. The framework surfaces invalid and ambiguous inputs before they ever reach a model.

The problem: the testing paradox in healthcare AI

Healthcare AI workflows are fragile. They connect disparate data sources, legacy EHR interfaces, and modern AI services. A failure at any handoff point can make a workflow unreliable, even if the underlying model is sound.

When I started architecting a solution, I ran into two roadblocks:

The privacy wall. You cannot legally or ethically use real Protected Health Information (PHI) to stress-test a new AI pipeline.
The context gap. Standard data quality tools check for null values or schema mismatches. They do not know that a human heart rate of 250 BPM is a data ingestion error, not a valid input. If bad data slips through, the model confidently generates a dangerous hallucination.

Teams need clear, testable evidence that a workflow can handle edge cases and invalid inputs before deployment, not after.

The architecture: a synthetic-first approach

Qualisera AI is a code-based prototype that explores repeatable quality checks in a controlled, zero-PHI environment. Three layers:

Layer 1 — Data ingestion

Instead of live hospital databases, the pipeline ingests from three legally safe sources:

Synthea™ for realistic, de-identified synthetic patient histories and vitals.
HAPI FHIR test servers for standards-compliant FHIR R4 resources (Patient, Observation, Bundle).
NIH Chest X-ray metadata for public, de-identified image finding labels.

Layer 2 — Clinical safety gates

Before any data proceeds, it passes a declarative validation layer built on great-expectations. Clinical boundaries are hardcoded into the CI/CD pipeline:

Heart rate must fall between 40 and 180 BPM.
Systolic blood pressure between 70 and 200 mmHg.
Birth dates cannot be in the future.

If a synthetic record violates a rule, the pipeline halts immediately.

Layer 3 — Mock inference and API testing

Validated data is sent to a deterministic, rule-based mock prediction endpoint built with FastAPI. This simulates a clinical AI model. We test payload validation, response shape, and performance — confirming the API returns 400 Bad Request for impossible vitals and responds within a strict <200ms latency SLA.

Validation and results

A QA framework is only as good as its coverage, so I built the suite around inspectable, scenario-oriented checks rather than a single pass/fail button. The current run collects 113 checks and reports 0 failures, split deliberately:

100 dataset scenarios exercise the three sources directly — Synthea (33), FHIR (34), NIH (33) — divided into 51 positive cases (valid inputs and expected behavior, e.g. a healthy profile returning a LOW risk, an in-range observation reported as in-range without inferring a disease) and 49 negative cases (invalid or incomplete input handling, e.g. a 250 BPM heart rate caught, a future birth date flagged, a unit mismatch blocking a misleading comparison, a missing reference range kept unassessed instead of called "normal").
13 supporting tests cover the machinery around the data — the mock endpoint's health and payload validation, response shape, latency, and the loader/dashboard plumbing that turns a run into a readable report.

The negative half is where the value sits. Proving the pipeline rejects a physiologically impossible vital, or refuses to silently label an observation "normal" when no source range was supplied, is exactly the behavior a reviewer looks for first. The positive half confirms the same gates don't false-alarm on clean data.

Limitations and scope

To be clear about what this prototype is not:

Not a diagnostic system — it does not diagnose, recommend treatment, or replace a clinician.
Not a trained clinical model — the prediction endpoint is a deterministic illustrative example.
Not end-to-end production monitoring — drift detection, outcome accuracy, alerting, and governance are not implemented as a closed loop.
Not full FHIR interoperability — selected resources are tested; real EHR conformance is not demonstrated.
Not a compliance certification — synthetic/test data does not establish HIPAA compliance on its own.

Next steps

The focus now is strengthening the evidence base: ensuring negative scenarios exercise the actual validators and API paths, bringing in qualified clinical stakeholders to review the hardcoded thresholds and escalation language, and defining data handling, access, and retention before considering sensitive environments.

These are software quality checks against test fixtures — not clinical validation, medical advice, or regulatory certification.

DEV Community: Moumita Ghosh