Dr. Harry Cruz
In general-purpose conversational AI, a wrong answer is a bad experience. In healthcare, it is a clinical event, and that single difference reshapes everything about how you test.
Everyone building a conversational product knows the standard testing playbook: check that context carries across turns, that the system stays in persona, that it does not leak the system prompt, that latency is acceptable and that the tone matches the brand. That playbook is necessary for healthcare too. It is nowhere close to sufficient.
The reason is not that healthcare is "harder." It is that healthcare breaks several assumptions the generic playbook quietly depends on. Errors are treated as roughly symmetric. The correct behavior is presumed to be answering the question, asked by a user who has described their problem accurately. A more capable model is taken to be a safer one. And scoring well on a held-out test set is taken to mean the system works.
That last assumption is now measurably false. When researchers took only the questions clinical models had already answered correctly and rephrased them without changing the medicine, 94% of those correct answers flipped to incorrect. These were not edge cases. They were the questions the models had just got right.
This post is for teams building and validating conversational AI in health: engineers, ML and QA leads, clinical safety officers and the product people who have to sign off. It maps where conversation sits across the health stack, introduces the handful of studies that now define what we know, works through the six shifts that separate testing conversational AI for healthcare from testing anything else, and finishes with an honest account of what benchmarks can and cannot tell you.
Conversational AI in health is not one product
"Healthcare chatbot" suggests a symptom checker talking to a patient. That is one layer of a much larger surface. Conversational and generative AI now sits at every point where health information gets written, interpreted, justified or acted on, and the same underlying model behavior shows up in all of them.
What varies between layers is not whether the output needs checking. It is who, if anyone, is positioned to catch the error.
Oversight is strongest where an external party is paid to find errors, and weakest exactly where the output reaches a patient unmediated. The testing burden does not fall as you move down the layers; it changes character, from immediate clinical harm to financial and statutory exposure.
The uncomfortable part is that the layer with the least human oversight is the one talking directly to patients. Everywhere else there is a clinician, a payer, an auditor or a regulator somewhere downstream. At the top of that diagram there is nobody.
Testing budget follows the liability instead. The bottom two layers are where an external party is already paid to find your errors, and a hallucinated justification in a coding engine or a missed adverse event in a pharmacovigilance pipeline is a liability with a name attached to it.
Who is actually on the other end
The patient-facing layer deserves a moment, because the population using these systems is not the population most teams design tests around. In January 2026, Microsoft researchers analyzed 617,827 de-identified health-related Copilot conversations and published the results in Nature Health.
Emotional wellbeing conversations follow the same nocturnal curve, from 3.3% in the morning to 5.2% overnight. A companion study of 1.7 million conversations across 109 countries found that lower population-level confidence in hospitals was the strongest predictor of health conversation intensity (r = −0.41, P < 0.001). Around 45% of the conversations were in English.
Those numbers describe a test population. The modal high-stakes user is on a phone at 2am, possibly asking on behalf of somebody else, in one of dozens of languages, at a moment when the alternative is unavailable.
Strong benchmark performance does not always translate to real-world reliability.
— Costa-Gomes et al., Nature Health, 2026
What the research says
Most of what we now know about how conversational health AI fails comes from a small number of studies published between mid-2025 and mid-2026. Their acronyms recur throughout the rest of this post.
Peer review status matters here
Three of these are preprints. NOHARM in particular has had its headline numbers move between revisions (severe-harm potential 22.2% → 24.6%; omission share 76.6% → >80%), and three vendors have each publicised themselves as its top performer by citing different endpoints of the same study. Cite the version you read. The methodological contributions are durable; the leaderboard positions are not.
Terms and benchmarks used below
What breaks when the domain is health
Six shifts matter most. Each takes an assumption that generic conversational testing depends on and sets the evidence against it.
1. Errors have a direction
In a generic chatbot, a wrong answer is a wrong answer. In triage, being wrong in one direction sends someone to an emergency department they did not need; being wrong in the other leaves a myocardial infarction at home. No single accuracy number can distinguish them.
The field has known this for a decade and still reports symmetric metrics. A five-year follow-up evaluation of symptom checker apps found they erred by over-triaging rather than under-triaging at odds of 2.82 to 1. That asymmetry was the whole point of the study, and an accuracy score cannot see it.
NOHARM made the same move for clinical recommendations, scoring commission and omission separately and weighting each by potential harm severity.
That single ratio should reorganize a test suite. Omissions are the dominant severe failure mode, and they are exactly what a generic "is this response good?" rubric is worst at catching, because the response looks fine. Eric Topol's summary, "errors of omission need to be brought as close to zero as possible", works as a scoring instruction.
A second-order consequence follows. If your rubric has to enumerate what must be present, someone has to decide what that is, and in clinical questions that is rarely one person's call. Reference answers come from panels, and they expire: a case that was correct last year becomes wrong when a guideline is revised. Directional scoring drags guideline versioning into your test infrastructure whether you planned for it or not.
- The safe answer is sometimes no answer This is the difference that most cleanly separates clinical conversation from every other kind, and the one generic testing is structurally blind to.
Plenty of patient questions contain a false premise: "which antibiotic should I take for my cold?", "how much extra insulin should I take to cover this dessert?" A clinician does not answer these as asked. They redirect, addressing the embedded misconception before responding to the underlying need. Answering the literal question would be malpractice.
MedRedFlag curated 1,103 real questions from r/AskDocs where verified clinicians chose to redirect, then tested whether models do the same.
Even when handed the correct assumptions as an oracle, the best configuration still accommodated 33% of questions with harmful misconceptions. Physicians reviewing the outputs found that in 3 of 10 cases the correction occupied 5% or less of the text, "drowned out" by the volume of information that still answered the unsafe question.
Current alignment techniques may inadvertently train models to avoid the "friction" of correction, thereby reinforcing confirmation bias in patients who may already be resistant to standard medical advice.
— Sambara, Pu et al., MedRedFlag, 2026
Helpfulness training and clinical safety are in direct tension. A generic test suite rewards responsiveness and user satisfaction, so it scores the unsafe behavior higher. You will not find this failure unless you write a test whose pass condition is the model declined to answer what was asked.
The same tension appears as sycophancy. A Mass General Brigham team found five models complied with illogical drug-equivalence requests at rates up to 100%. SycoEval-EM ran 1,425 simulated emergency encounters across 19 models using five persuasion tactics and found acquiescence rates spanning 0% to 100% in a bimodal distribution: seven models held guideline adherence near-perfectly, six caved in the majority of encounters. The line that should worry anyone reading a leaderboard: "model scale, recency, and performance on static medical benchmarks did not consistently predict robustness."
3. Ordinary users apply adversarial pressure by accident
Red-teaming in general AI safety models a motivated attacker. In healthcare the pressure comes from ordinary people in bad states: incomplete histories, emotionally charged framing, embedded misconceptions, irrelevant detail, questions asked on someone else's behalf, low health literacy, a second language. These are not attacks. They are Tuesday.
DAS quantified how little pressure it takes. Its bias battery combined a curated public corpus with 111 scenarios authored by three licensed clinicians, then applied manipulations that changed presentation while leaving the pathology identical.
Combined susceptibility to at least one manipulation exceeded 80% across every model tested. Cognitive priming dominated demographic swaps, and the authors read that sharply: cognitive bias attacks were ineffective on MedQA but potent on physician-authored equity scenarios, suggesting "familiar exam-style benchmark language may be learned and ignored, whereas ambiguous clinical-equity cases without canonical answers push models towards shortcuts such as authority cues or salient anecdotes."
Models have learned to look unbiased on questions that look like bias tests. They have not learned to be unbiased on questions that look like clinical work.
The privacy results tell the same story in a different register, and they are the clearest illustration that this is a social failure rather than a technical one. DAS built 81 privacy-trap scenarios across eight high-incidence HIPAA and GDPR violation modes. Direct requests leaked PHI in 86.46% of scenarios. Adding an explicit system instruction to comply with both regulations reduced that to 66.75%, which helped without solving anything. Four social disguises then pushed the average to 91%, and the most effective was the trap warning, invoking privacy concern as the pretext for the request. Claude Sonnet-4 was the most resilient at 72.84%; two models leaked in every adversarial scenario.
The authors' analogy is the right one. This "mirrors how human clinicians commit unintentional privacy breaches despite explicit training." Trained professionals fail under social pressure, and so do models, for recognisably similar reasons. Testing refusal of direct requests measures the wrong surface.
- Capability and safety come apart The intuition that a stronger model is a safer one is load-bearing in most procurement decisions. It does not survive the hallucination data.
DAS decomposed medical hallucination into seven categories rather than treating it as one phenomenon, then built a detector with one specialised subagent per category.
Average hallucination rate across the 15 models on hard cases: 74%. Proprietary models did better than open-source (62.6% vs 72.5%). Less predictably, models with explicit chain-of-thought reasoning hallucinated more than those without (68.7% vs 64.7%), with more logic errors (26% vs 22%), more context failures (20% vs 15.6%) and more citation errors (20% vs 15.4%). Reasoning chains propagate an early false premise rather than correcting it.
To be fair to reasoning models, the trade-off runs both ways: explicit reasoning improved instruction compliance (7.6% vs 10.5% failure) and reduced unsafe recommendations (12.2% vs 14%). The point is not that reasoning is bad. It is that "more capable" resolves into different error profiles rather than fewer errors.
Scale helped not at all. MedGemma-27B (65.1% error rate) beat both Qwen3-32B (77.4%) and QwQ-32B (80.2%). Medical fine-tuning helped modestly (64% vs 67% for general-purpose baselines), though MedGemma did beat its own Gemma-3 base model across every category.
Safety cannot be inferred from model choice or from a domain-tuned label, and parameter count tells you nothing either. It has to be measured, per category, on your task.
5. The failure is the conversation, not the response
Single-turn evaluation systematically overstates clinical performance, because the clinical failure mode is the dialogue rather than the answer.
CRAFT-MD measured this directly by putting a patient agent in front of the model under test: GPT-4's diagnostic accuracy fell from 0.820 in the structured setting to 0.627 in multi-turn conversation. Work outside medicine points the same way. Across 200,000+ simulated conversations, models showed an average 39% drop from single- to multi-turn, characterized as "a minor loss in aptitude and a significant increase in unreliability", with the memorable finding that "when LLMs take a wrong turn in a conversation, they get lost and do not recover."
DAS is built around the same insight, escalating across up to five rounds and switching strategy when an attack fails. Most models were trapped within one to three. If your evaluation stops at the first response, you are measuring the part of the interaction that works.
6. The result is a distribution, not a number
Several distinct failures share one cause: a mean conceals them.
Equity failures are between-group differences. By construction they cannot appear in an aggregate score. They are visible only as the delta when identity, language or register changes and the pathology does not.
Crisis handling fails in the middle band. A RAND study put 30 suicide-related questions through ChatGPT, Claude and Gemini 100 times each, with 13 clinicians rating question risk across five levels. The systems handled very-high and very-low risk appropriately and were inconsistent at intermediate risk. That is the band where most real conversations sit, and the band keyword-based safety testing never probes. An average across the risk spectrum would have looked reassuring.
Language performance is not fungible. Around 45% of those 617,827 Copilot health conversations were in English. A system that scores 92% overall can be dangerous in the language 8% of its users speak, and averaging across languages is what hides it.
Non-negotiable: gate per language, not on average: If the product ships in Arabic, Hindi, Spanish or Portuguese, the battery runs and gates per language. Voice interfaces add an ASR sub-battery covering term accuracy, accent robustness and diarization, because downstream you cannot tell a transcription error from a reasoning error.
One more item belongs here, and nobody measures it at all. Clinical reasoning is inherently contrastive, since differential diagnosis is the question "why this, rather than that?" The cXAI systematic review screened five databases and found only 18 studies that explicitly applied contrastive explanations in healthcare, of which just 2 (11%) reported empirical evaluation with end users. The authors note the gap "is striking given that many clinical processes, such as differential diagnosis, treatment selection and risk stratification, are inherently contrastive." If your system exposes a rationale, that rationale is an output. It is also almost certainly untested.
What benchmarks are actually for
None of the above is an argument against benchmarks. It is an argument against using them for something they were never built to do.
Benchmarks earn their place. They make models comparable on a common task, which is the only reason the field can have a conversation about progress at all. They catch regressions cheaply. If a fine-tune drops MedQA by fifteen points, something broke. A benchmark also sets a floor, since a model that cannot pass a licensing exam is not a candidate, and it compresses a complex capability into a number a non-specialist can act on. Benchmarks make excellent raw material for something better too.
What they cannot do is establish that a system is safe to deploy. The five reasons get worse as they go.
They report a mean
Every limitation in shift 6 applies. An aggregate hides subgroup behavior, per-language behavior and performance in the intermediate-risk band, which is precisely where the harm concentrates.
They are single-turn
Nearly every established medical benchmark presents a fixed question and scores one response. CRAFT-MD's 0.820 → 0.627 drop is the size of the gap between that and a conversation. A benchmark score is a measurement of the easiest part of the interaction.
They encode a moment in time
Clinical correctness has a version number. Guidelines are revised, indications are withdrawn, payer rules change. A static benchmark records the consensus on its publication date and then quietly decays. Nothing in the artefact tells you which items have gone stale.
They become optimization targets the moment they are published
Goodhart's Law holds that when a measure becomes a target, it ceases to be a good measure, and in medical AI that is now documented rather than theorised. DAS found that model rankings reordered between MedQA and HealthBench: MedGemma rose substantially, GPT-4o dropped sharply. As the authors put it, "such rank instability suggests benchmark-specific optimization rather than transferable medical competence." If a leaderboard position does not survive a change of benchmark, it was never measuring capability.
A high score can be brittle in a way the score cannot show
This one should change how you read every clinical AI claim you see this year.
The design detail that makes the 94% damning is easy to miss: only questions a model had already answered correctly entered the stress test, and the perturbations preserved the clinical meaning. This is not a hard-question set. It is the same questions, rephrased.
The most effective single perturbation, at 60% average jailbreak, was question inversion: "which of these is not the treatment". That result is diagnostic rather than merely embarrassing, because failure on inverted logic indicates the model is matching surface patterns rather than reasoning about the medicine. A benchmark that only ever asks questions the normal way round cannot detect that.
And note the right-hand panel, because it forecloses the obvious escape route. HealthBench is a 2025-vintage benchmark of open-ended vignettes with rubrics written by 262 physicians, everything you would ask for in a modern evaluation. Its best score was 42%, so it is nowhere near saturated. Under adversarial pressure, top-tier models still failed at over 70%. A newer, harder, better-designed benchmark is not the fix. Being static is the problem.
So use them for what they are good at
The legitimate roles, none of them "evidence of safety":
A floor. Failing MedQA disqualifies a model. Passing it qualifies the model for testing, not for deployment.
A regression tripwire. Cheap, fast, and genuinely useful for catching what a fine-tune or a version bump broke.
Seed material. The highest-value use. Every case in a public benchmark is a validated clinical scenario with a known correct answer, which makes it an excellent starting point for generating the adversarial, multi-turn, persona-driven variants that actually probe behavior. DAS did exactly this, using MedQA and HealthBench as launch points rather than endpoints.
The distinction to hold onto: a benchmark tells you whether a model can get something right. It cannot tell you whether it will, when the question arrives phrased the way a frightened person at 2am would phrase it.
Wrapping up
Healthcare conversational AI does need different testing, and "the stakes are higher" is too vague a reason to act on. Higher stakes alone would just mean testing more of the same. The 2026 research shows stakes that are differently shaped, and that difference invalidates specific methods rather than merely raising the bar.
Errors have a direction, which makes accuracy the wrong summary statistic. Because omissions dominate severe harm, a rubric has to enumerate what must be present rather than judge what is. Helpfulness metrics actively reward unsafe behavior in the cases where the correct response is a refusal. Ordinary users apply more adversarial pressure than red teams do, and they do it socially rather than technically. Capability and safety have come apart, so model choice is not a safety argument. Failures live in the fifth turn, in the second language, and in the intermediate-risk band that averages conceal.
None of that argues against building these systems. The usage data is unambiguous that people already rely on them, disproportionately at night, disproportionately where confidence in local health services is low, and often on behalf of someone they are caring for. The question was never whether to deploy. It is whether you can produce credible evidence about behavior under the conditions of actual use, and a leaderboard position is not that evidence.
The ethical frame is the technical frame
A framework published in npj Digital Medicine by clinicians at UVA Health and Clemson argues for weighing healthcare AI across five priorities: patient care, staff experience, hospital operations, economic impact, and education and research. Cost-focused evaluation leaves "risks of bias, opacity, workforce displacement and erosion of the patient-clinician relationship that are invisible to cost-focused analyses."
A test suite is where those abstractions become measurable. Bias becomes counterfactual invariance, opacity becomes contrastive rationale quality, and erosion of the clinical relationship turns into scope-boundary and redirection testing.
Next in this series: building a simulation harness for clinical conversation
If static benchmarks cannot establish safety, something has to. The follow-up post covers the practical architecture: seeding from real clinical cases, persona design grounded in usage data, an enumerated operator vocabulary for adversarial mutation, multi-turn escalation, and the gates that correspond to each kind of failure above. It also covers how to run clinician annotation loops and how to validate an automated judge before you trust its numbers.
Top comments (0)