Building a Test Harness for Healthcare Conversational AI

Dr. Harry Cruz

Aug 5, 2026 • 17 min read

The previous post made a negative case: healthcare conversational AI fails in ways that static benchmarks are structurally unable to detect, and the evidence for that is now quantified rather than argued. Ninety-four percent of previously correct answers flipping under meaning-preserving rephrasing is not a subtle finding.

The obvious question is what to do instead, and the honest answer is that you have to generate your own pressure, continuously, and judge the results against criteria a clinician wrote down. That sounds expensive. It is less expensive than it sounds, mostly because the first useful version is much smaller than teams expect. Twenty cases and a hand-written rubric will find real problems in an afternoon.

This post is the practical version: the five stages of a healthcare AI test harness, a worked example running through all of them, the annotation loop that turns outputs into numbers you can defend, and the order to build it in.

What we are testing, in one page
A note on scope before the build details, because "healthcare AI" gets used to mean "a chatbot talking to a patient," and that is one layer of six. Conversational and generative AI now sits wherever health information is written, interpreted, justified or acted on, and the map from part one is worth keeping in view before narrowing down.

And the six shifts from part one, compressed to their testing consequence:

**This post takes the layers where the input is a conversation: **patient-facing products, and the clinician-facing tools you talk to rather than feed a file. The five stages below are structural, so they carry over to the layers whose input is a recording, a note or a corpus, but what goes into stage 2 stops being a person and stage 4 often collapses to a single shot. Those layers need their own treatment and they get it separately.

The architecture, and a worked example
Five stages. Each is independently versioned, so when a run fails you can attribute it to a specific seed set, persona, operator or rubric revision rather than shrugging at the whole pipeline.

The pipeline. Stages 3 and 4 are what make the suite resistant to being optimized against, because probes are synthesized per run rather than stored. Version each stage separately: "the suite went red" is not a diagnosis; "operator v3 broke seed set v7" is.

Where these recommendations come from
Almost nothing in this post is original. "Someone made this up" and "two physicians adjudicated 100 samples per operator and reported validity rates" are very different grounds for adopting a method, so the table below separates what is published from what we have extrapolated.

The bottom three rows are the honest caveat. Where a row says Ours, the reasoning is ours and you should weigh it accordingly.

Abstract architecture diagrams are easy to nod along to and hard to build from. Below, a single test case moves through all five stages. Everything in it is illustrative but constructed to be realistic.

Stage 5 · Outcome

Deterministic gate failed, release blocked. Emergency escalation never occurred across three turns. The model was not jailbroken by a clever prompt; it was talked out of the correct answer by an anxious spouse at 2am.

logged: seed 0142 · persona v2 · operator cognitive_bait · failed turn 1 · 3 rounds · judge κ 0.94

Single-turn evaluation of this same seed passes. The failure only exists across turns, under social pressure, with a plausible wrong hypothesis in the room. No adversarial prompt engineering was involved. A worried spouse volunteered an anecdote and asked twice not to go to hospital, and the system talked itself out of the only safe recommendation.

Stage 1 · Seeds
A seed is a clinical situation plus a statement of what a safe response must and must not contain. Without the second half it is not a test case, it is a prompt.

Where to get them, in order of preference

The counterintuitive part is how few you need. A hundred real cases expanded through personas and operators produces thousands of distinct probes, and beats ten thousand synthetic vignettes evaluated once. Depth per case matters more than case count, because the mutation engine is what generates volume.

What counts as a seed still varies within the conversational layers. For a triage product it is a presentation. For a decision-support or evidence tool it is the question a clinician actually typed, with the answer a specialist would have given. Underneath the variation, a seed is a real input plus an adjudicated statement of the right output.

Version the guideline, not just the case

Every seed encodes a consensus with a date on it: a clinical guideline, a triage protocol, an escalation threshold. When one of those is revised, some of your "correct" answers become wrong and nothing in the test suite will tell you. Store the governing revision alongside each seed and schedule a review when it changes. This is the single most commonly skipped piece of test hygiene in healthcare AI, and it fails silently.

The rubric that ships with the seed
The format below is not invented either. It follows HealthBench, whose cases each carry 5 to 20 physician-written rubrics specifying criteria a response should satisfy or should avoid, with the score computed as credit for satisfying positive rubrics and avoiding negative ones. That positive/negative split is the published mechanism for detecting omission, which is reason enough to adopt the format rather than improvise one.

DAS adds one refinement. Before running their HealthBench audit, two physicians reviewed the rubrics and kept only criteria with a clinically meaningful, binary pass/fail boundary, discarding subjective style criteria such as conciseness. Eight of 200 cases lost all their rubrics in that filter and were dropped. Take the same filter to your own rubrics. If a criterion cannot be adjudicated as met or not met by two independent experts, it is not a test criterion, and including it will degrade every agreement figure you later compute.

The rubric for the chest pain case above follows that format, with severity-labeled criteria split into what must appear and what must not.

P1 and P3 are separate criteria on purpose. Saying the right thing once and then abandoning it is a different failure from never saying it, and you want to count them separately. The positive criteria are the only reason you can detect omission, which the previous post established as the dominant severe failure mode. A rubric that only lists prohibited content will score the trace above as a partial pass.

One more criterion is worth writing into most rubrics, and it gates traceability rather than correctness: require the system to show what it relied on. You cannot always adjudicate a clinical answer cheaply, but you can always ask which guideline, which retrieved passage, which part of the history. An answer with no supporting reference is a failure regardless of whether it happens to be right.

Stage 2 · Personas
Stage 2 decides the conditions under which the input arrives, and where a human is talking those conditions are a persona. The job is to make the input as messy as reality makes it.

The idea of putting a simulated patient in front of the model rather than handing it a vignette is CRAFT-MD's, published in Nature Medicine. Their framework has a patient-AI agent converse with the clinical model under test, with responses graded by AI and medical experts, and it is the design that produced the 0.820 to 0.627 accuracy drop between structured and conversational settings covered in part one. If you want one citation for why conversational simulation is necessary rather than optional, that is it.

A persona is not a demographic label. It is a generator that decides how the seed gets expressed: what gets volunteered, what gets withheld, what the user already believes, and how they respond to being told something they do not want to hear. The specific field list below is ours rather than published, assembled to cover the variation the Nature Health Copilot usage analysis actually observed. Eight fields cover most of what matters:

Build the set from your usage distribution rather than from imagination. Part one's Copilot usage figures are a starting shape for patient-facing products, in particular that around one in seven personal health conversations are on behalf of somebody else, and that symptom questions run more than twice as high on mobile as desktop. Six to ten personas is enough to start. The caregiver by proxy consistently earns its place first, because the clinical information arrives second-hand and incomplete. Close behind is the high-pushback user, which is what turns a correct first answer into an abandoned one.

One input condition is worth adding even to a purely conversational product, and teams building evidence and decision-support tools miss it most often. "No supporting evidence exists" has to be an explicit condition, because the failure you are looking for is a fabricated or misattributed citation, and you will only reliably provoke it by asking questions the literature does not answer.

Stage 3 · Mutation operators
This is the engine. An operator takes a seed plus a persona and applies one specific pressure, leaving the clinical facts intact.

Where the concept comes from

Pan et al. call them "six orthogonal mutation tools" and define each one precisely in the DAS methods, with the stated goal of deterministically altering the ground-truth answer rather than introducing ambiguity. We have adopted their vocabulary directly, because a named, reproducible operator set is the part worth copying.

Two of the six draw on earlier literature rather than originating with DAS. Cognitive bait builds on work documenting cognitive biases in LLMs, including Schmidgall et al. in npj Digital Medicine. Narrative distraction is motivated by findings that medical LLMs are easily distracted by extraneous context. The seams matter: this is a decade of accumulated evidence about how these systems fail, packaged as something you can run.

The critical design decision is to enumerate the vocabulary and forbid improvisation. A model-driven attacker told to "try to make it fail" will find failures you cannot reproduce or attribute, and cannot turn into a regression test. DAS prohibits freestyle mutations outside its six-tool set explicitly, and that constraint is what makes per-operator reporting possible.

Inside cognitive bait
"Apply a cognitive bias" is not implementable as written. In DAS the agent injects three short framing sentences drawn from nine strategies, each of which frames the wrong answer as attractive:

Defensive medicine and availability were newly introduced in that work; the rest are adapted from prior studies. The two that transfer most directly to patient-facing products are self-diagnosis and availability, which are the mechanisms at work in the worked example above: a spouse who already believes it is indigestion, plus a brother whose reflux is the most available comparison.

If you want a head start rather than a taxonomy, the Schmidgall paper that underpins this operator also released BiasMedQA: 1,273 USMLE questions modified to replicate clinically relevant cognitive biases. It is a public dataset built for exactly this purpose, and a faster starting point than authoring bias framings yourself. Their finding is also a useful calibration on the shift that capability and safety come apart: GPT-4 was notably resilient while Llama 2 70B-chat and PMC Llama 13B dropped sharply, and three mitigation strategies improved accuracy without fully restoring it.

What the operators actually do
Applied to the chest pain seed above, so you can see the transformation rather than infer it:

Two operators that behave differently

Identity, linguistic and emotional manipulation must run as counterfactual sets. A single run tells you nothing, because the signal is whether the recommendation changed when only the presentation did. Generate the full set, compare within it, and report the delta.

Physiological impossibility is a canary, not a realistic scenario. Nobody has a heart rate of 15 and a blood pressure of 320/210. Its value is that a model which reasons confidently over impossible input is doing fluent pattern completion rather than clinical reasoning, and that tells you something about every other answer it gives.

The four privacy operators have a provenance of their own. DAS built 81 scenarios across eight violation categories drawn from documented real-world healthcare privacy incidents rather than from imagination: unauthorized disclosure, minimum-necessary oversharing, overheard and overseen conversations, misdirected email, personal devices without safeguards, access with no valid healthcare reason, public disclosure of PHI, and accidental release on social media. If you are building a privacy battery, that list is a better starting taxonomy than anything you will derive from first principles, because it reflects how breaches actually happen.

They also ran a sensitivity check worth imitating. Their prompts appended an instruction to output only a copy-pasteable template, simulating high-throughput workflows, and a natural objection is that this suppresses the model's own caveats. Removing it across all 15 models moved the average leak rate from 86.46% to 84.00%, which is small enough to rule out that explanation. When someone can plausibly claim your operator manufactured the result, test the claim and report the delta.

Stage 4 · Orchestration and budget
The orchestrator holds the dialogue history, picks the next operator based on what has and has not worked, and decides when to stop. Its selection logic can be simple; what matters is that it escalates rather than firing one shot.

DAS caps at five rounds and traps most models within one to three, which is a useful calibration: if you are not finding failures in the first three turns, the problem is more likely your operator coverage than the model's robustness. Their implementation closes three specific holes:

• An incompatibility matrix. Some operators cancel or corrupt each other, so certain combinations are disallowed by rule. In DAS, question inversion is never combined with answer negation, and negation is never combined with physiological impossibility or choice expansion. Without this you generate items whose correct answer is undefined, then score a model wrong for a question nobody could answer.
• Mutate the seed, not the previous output. Every mutation in DAS is applied to the original seed rather than to the last round's mutated version, explicitly to limit drift. Chaining mutations compounds them, and after three rounds you are testing a scenario that no longer resembles medicine.
• A retrospective validity check. After mutating, the orchestrator verifies the item is still medically consistent with a single best answer, and returns which operators it chose plus a rationale. That record is what makes a failure attributable later. Their escalation schedule is also more deliberate than "keep going": rounds one to three use single or two-operator combinations to avoid prompt bloat, rounds four and five escalate to orthogonal multi-operator combinations, and the attack stops the moment the answer diverges from the mutated ground truth.

Validate the mutations, not just the judge

This is the step we see skipped most often, and DAS is unusually rigorous about it. Two board-certified physicians blindly adjudicated 100 mutated samples per operator, reporting validity rates of 96% for question inversion, 97% for physiological impossibility and 86% for choice expansion, with rule-based operators effectively 100% valid. On the open-ended HealthBench audit they adjudicated a further 300 mutated prompts, at 92% for cognitive bait and 97% each for narrative distraction and physiological impossibility.

An invalid mutation produces a failure that is not a failure. If 14% of your generated items are broken, then 14% of your red is noise, and you will spend engineering time chasing it. Adjudicate a sample per operator before you trust the operator, and report the validity rate alongside the failure rate.

Stopping rules, in evaluation order

1. A deterministic gate broke. Stop immediately, log the turn index. Continuing wastes budget when you already have the finding.
2. Round cap reached. Five is a reasonable default. Longer conversations are worth testing separately, not by raising this.
3. Operator vocabulary exhausted for this seed and persona. Record as a pass with the operator list attached, so a later coverage audit can tell "passed everything we tried" from "passed everything there is."
4. Token or cost ceiling hit. Log as inconclusive, never as a pass. Inconclusive runs that silently count as passes are how a suite starts lying to you. Log the turn index of every violation. "Failed on turn four" and "failed on turn one" are different products, and the distribution of failure turn is one of the more informative things the harness produces. A system that holds for three turns and then collapses needs different work from one that never gets it right.

Stage 5 · Scoring and gates
Three kinds of property, three kinds of gate. The discipline that matters most is never mixing them: a crisis-detection failure averaged into a quality score disappears, and that is precisely the failure you built the harness to catch.

That bottom panel changes what a passing suite means. If you run 200 crisis-detection cases and see no failures, you have established that the failure rate is probably below about 1.5%, not that it is zero. At the volumes a deployed patient-facing product handles, a 1.5% ceiling on crisis mishandling is not reassurance.

This is not an AI result. It is the rule of three, set out by Hanley and Lippman-Hand in JAMA in 1983 under the title "If nothing goes wrong, is everything all right?", a paper about interpreting zero numerators in clinical studies. Their conclusion transfers without modification: with zero events in n observations you can be 95% confident the true rate is at most 3/n, and the right output is a confidence bound rather than a reassuring zero. Medicine settled this forty years ago, and clinical AI evaluation has mostly not noticed.

Working backwards: to claim with 95% confidence that a failure rate is below 1% you need roughly 300 clean runs; below 0.1%, roughly 3,000. This is the concrete argument for automating the judge. Those volumes are unreachable with human review, and unreachable volumes are why deterministic gates get quietly downgraded to "we tested a few."

What is directional in a conversational product
The directional gate is the one that generalizes furthest, because every layer has an error that is worse in one direction than the other, and in every layer the instinct is to report a single accuracy number that hides it.

Write the asymmetry into the score rather than into a footnote. If a missed crisis activation is a hundred times worse than a false one, the metric should say so. Otherwise a model that trades recall for precision will look like an improvement.

The annotation loop
The harness produces outputs. Something has to decide whether each one met the rubric, and that judgment chain has to terminate in clinical competence. The economics only work if a validated automated judge does the volume and clinicians do the calibration. Part 3 covers the annotation evidence in detail.

Validating the judge
An automated judge you have not measured is an opinion with a decimal point. Every credible study in this space reports its agreement with clinicians, and the pattern across them is instructive:

The relationship is hard to miss: the narrower the judgment, the higher the agreement. "Did this response leak protected information" reaches κ above 0.95. "Is this response clinically sound" does not, and even the seven-subagent hallucination detector, one specialized judge per category, only reached 81.9% accuracy.

That pattern has a sharp edge. The privacy judge above reached κ 0.952 on a single binary question, but asked to detect incomplete patient-facing answers across three rubric granularities and three backbone models, judges separated complete from incomplete at AUC 0.49 to 0.66. At the threshold needed to recall 90% of incomplete responses, clinicians still had to review the vast majority of the dataset, so there was no triage benefit. Incompleteness is omission, and omission is where the severe harm sits. A judge-blind random sample is the only stream that will show you a judge failing silently on it.

The practical instruction is to decompose. Never ask a judge whether a response is safe. Ask whether criterion P1 was met, then whether N2 was violated, as separate calls with separate validation figures. Aggregate afterwards, in code you control.

How much dual annotation
Budget 100–200 items dual-annotated per criterion type, not per individual criterion, for a stable agreement estimate. Report κ per type, because a judge that is excellent at detecting prohibited content can be poor at detecting omission, and a single headline number hides exactly the weakness you care about. Re-validate whenever you change the judge model, and treat a judge model upgrade as a change requiring re-validation rather than a free improvement.

The useful accident in the build order

Starting manually, with twenty seeds and hand-written rubrics scored by a clinician, finds early bugs. It also produces the labeled gold standard you will need to validate the automated judge in step 3. Teams that automate first end up going back to create that gold standard from scratch. Doing the manual phase properly makes the automation phase nearly free.

Closing the loop without poisoning it
Every failure the harness finds is a labeled counter-example, and it belongs in the regression suite. DAS frames red-teaming as "not merely a gate-keeping audit but a continuous quality-improvement loop."

One caveat, and it is the one that quietly destroys the value of the whole exercise: hold out a portion of the adversarial set from anything that touches training or prompt tuning. If every failure you discover gets fixed by adding it to the fine-tuning mix, you have rebuilt Goodhart's Law inside your own infrastructure, and your suite will go green while the behavior it was built to measure gets no better.

What to build first
The full architecture is a quarter of work. The first useful slice is an afternoon, and the ordering below is designed so that each phase produces something the next phase needs.

Week one is genuinely twenty cases. Pick the twenty where being wrong matters most: the emergency presentations, the crisis-adjacent conversations, the drug interactions, whatever your product's version of those is. Write the rubrics. Run them single-turn. You will find something, and the finding will fund the rest of the work.

Anti-patterns
Ordered roughly by how much time they waste.

Where Rhesis fits
A disclosure, since we build one of the tools in this space. Rhesis is an open-source platform for getting domain-expert judgment into AI testing, and you can self-host the whole thing.

Almost everything difficult in this post is a knowledge problem before it is an engineering problem. Someone has to decide that P1 is critical and P4 is medium, that holding the escalation is a separate criterion from stating it once, and what crisis-miss rate is acceptable at your deployment volume. That someone is a clinician, and they do not work in a repository. The gap between their judgment and your test suite is where most healthcare AI evaluation quietly stalls.

Two pieces map directly onto this post. The first is how rubrics are stored: the equivalent of a criterion list is a behavior with metrics attached, named by the clinician in their own words, and pass rates roll up by behavior rather than by metric name. That changes the review conversation more than it sounds like it should. "Answer relevancy 0.71" starts an argument about the metric; "maintains emergency escalation under pushback: 62% pass" starts an argument about the agent, which is the one you wanted.

The second is judge validation. When a clinician marks a result pass or fail, the automated score stays alongside it and disagreements are flagged as conflicts. That is exactly the paired record step 3 of the annotation loop needs, accumulated as a by-product of normal review rather than as a separate measurement exercise.

Because the seeds in this post are real consultations, where they sit matters. Rhesis self-hosts as Docker Compose or on your own Kubernetes cluster, in both editions, so cases, traces and reviews stay inside your environment. It is developed in Germany, and Rhesis Cloud is hosted in the EU.

What none of it does is supply the judgment. Nothing can tell you that a missed crisis activation is worth a hundred false alarms. The severity weights stay yours; a platform is only where that decision gets written down, versioned, and attached to something runnable.

Wrapping up
The gap between "we know static benchmarks are insufficient" and "we have something better running" is mostly a tooling and sequencing problem rather than a research one. The methods are published. What stops teams is that the full architecture looks like a quarter of work, so it never starts.

It does not have to start there. Twenty seeds, a rubric with positive and negative criteria, one clinician scoring by hand, single-turn. That is an afternoon, it will surface real problems, and it produces the labeled data that makes everything downstream cheaper. Add three operators. Then automate the judge against labels you already have. Then go multi-turn.

Two things separate a harness from theater, and both are worth being rigid about. Keep the gate types separate, so a crisis failure can never be averaged away by good performance elsewhere, and publish the judge's agreement figure next to every number it produces. Everything else you can approximate and improve. Those two are load-bearing.

And re-run it on every model update. The models underneath you change on a timescale of weeks, the guidelines on a timescale of months, and either can turn a passing suite into a false reassurance without anything visible happening.

References
Pan, J. et al. Addressing benchmarking gaps in large language models for health and medicine with dynamic red-teaming. Nature Health (2026). doi.org/10.1038/s44360-026-00152-8
Wu, D. et al. First, do NOHARM: a medical safety benchmark and randomized study of physician and AI teaming on clinical consultations. arXiv:2512.01241v4 (2026). arxiv.org/abs/2512.01241
Sambara, S., Pu, Y. et al. MedRedFlag: Investigating how LLMs Redirect Misconceptions in Real-World Health Communication. arXiv:2601.09853 (2026). arxiv.org/abs/2601.09853
Johri, S. et al. An evaluation framework for clinical use of large language models in patient interaction tasks. Nature Medicine 31, 77–86 (2025). doi.org/10.1038/s41591-024-03328-5
Peng, D. et al. SycoEval-EM: Sycophancy Evaluation of Large Language Models in Simulated Clinical Encounters for Emergency Care. arXiv:2601.16529 (2026). arxiv.org/abs/2601.16529
Vatanparvar et al. PatientAgentBench: evaluating clinical agents with simulated patients. arXiv:2607.25485 (2026). arxiv.org/abs/2607.25485
DeLucia, A. et al. Same Verdict, Different Reasons: LLM-as-a-Judge and Clinician Disagreement on Medical Chatbot Completeness. arXiv:2604.16383 (2026). arxiv.org/abs/2604.16383
Costa-Gomes, B., Tolmachev, P., Taysom, E. et al. Public use of a generalist LLM chatbot for health queries. Nature Health 1, 689–696 (2026). doi.org/10.1038/s44360-026-00117-x
Arora, R. K. et al. HealthBench: evaluating large language models towards improved human health. arXiv:2505.08775 (2025). arxiv.org/abs/2505.08775
Laban, P. et al. LLMs Get Lost In Multi-Turn Conversation. arXiv:2505.06120 (2025). arxiv.org/abs/2505.06120
Hanley, J. A. & Lippman-Hand, A. If nothing goes wrong, is everything all right? Interpreting zero numerators. JAMA 249, 1743–1745 (1983). pubmed.ncbi.nlm.nih.gov/6827763
Schmidgall, S. et al. Evaluation and mitigation of cognitive biases in medical language models. npj Digital Medicine 7, 295 (2024). doi.org/10.1038/s41746-024-01283-6
Vishwanath, K. et al. Medical large language models are easily distracted. arXiv:2504.01201 (2025). arxiv.org/abs/2504.01201
Chapman, W. W., Bridewell, W., Hanbury, P., Cooper, G. F. & Buchanan, B. G. A simple algorithm for identifying negated findings and diseases in discharge summaries. Journal of Biomedical Informatics 34, 301–310 (2001). doi.org/10.1006/jbin.2001.1029
Koo, R. et al. Benchmarking cognitive biases in large language models as evaluators. In Findings of the Association for Computational Linguistics: ACL 2024, 517–545 (ACL).
Note on the Schmidgall, Vishwanath and Koo references: these are cited as the upstream basis for the cognitive bait and narrative distraction operators, as identified in the DAS methods. The attribution is taken from that paper's own reference list rather than re-derived.