I Benchmarked 3 LLMs on Clinical Care-Management Tasks — They Prescribe the Drug and Skip the Safety Check

Habib Ur Rehman — Fri, 09 Oct 2026 15:41:26 +0000

This is a submission for the Kaggle Benchmarking Challenge.

I Benchmarked 3 LLMs on Clinical Care-Management Tasks — They Prescribe the Drug and Skip the Safety Check

Everyone picks an LLM for healthcare work based on MMLU scores and marketing copy. I wanted to answer a narrower question: which models can actually do care-management work — read a patient case and produce a care plan that is not just fluent, but complete and safe?

So I built CareManage-Bench: 24 synthetic clinical cases across 3 task families, each with ground-truth checklist assertions. I ran it against 3 models on Kaggle Benchmarks. The models aced the scary-sounding tasks and failed the boring ones — and the failure mode is the interesting part.

What I benchmarked

CareManage-Bench — 24 tasks in 3 families, all cases fully synthetic (no real patient data anywhere):

Medication safety review (8 tasks) — given a patient's med list, flag the planted danger: a drug interaction, a contraindication, a dosing error. Scored on whether the model catches the specific planted hazard.
Care-plan generation (8 tasks) — given a patient summary, write a structured care plan (assessment, medications, lifestyle, referrals, monitoring, education). Scored against a must-include checklist per case: e.g., a new-AFib case must mention anticoagulation and bleeding-risk assessment; an osteoporosis case must mention bisphosphonates and dental review before starting them.
Triage prioritization (8 tasks) — rank a queue of cases by urgency in an exact format. Scored on exact ranking match.

Why these three? They mirror what a care-management team actually does all day: don't harm (med safety), do everything needed (care plans), do the urgent thing first (triage). And the scoring is deliberately harsh: checklist assertions, all-or-nothing per task. A beautiful plan that misses one safety item scores zero — because in clinic, that's what a miss costs.

Models tested

Gemini 3.7 Flash (google/gemini-3.7-flash) — Google's fast flagship; the baseline to beat
Gemma 4 31B (google/gemma-4-31b) — open-weights model; can it hang with the flagships on clinical reasoning?
GPT-5.4 Mini (openai/gpt-5.4-mini-2026-03-17) — OpenAI's efficient tier; the cost-effective deployment candidate

Same harness, same prompts, same assertions for all three. No prompt tuning per model — I wanted to measure the models, not my prompt engineering.

Findings

The scoreboard

| | Med safety (/8) | Care plans (/8) | Triage (/8) | Total (/24) |
|---|---|---|---|
| Gemini 3.7 Flash | 8 | 7 | 8 | 23 |
| Gemma 4 31B | 8 | 5 | 8 | 21 |
| GPT-5.4 Mini | 8 | 7 | 8 | 23 |

67 out of 72 overall (93%). Two of three families were perfect across all models. The entire benchmark was decided by 5 failed care-plan tasks.

Finding 1: Every model aced the "scary" tasks

All three models went 24/24 on medication safety and triage. Every planted drug interaction caught. Every contraindication flagged. Every triage queue ranked exactly right. If you only benchmarked danger-detection, you'd conclude these models are interchangeable and ready to deploy.

Finding 2: The depression-screening blind spot (all 3 models)

The post-MI case (68-year-old woman, NSTEMI 2 weeks ago, stent, EF 40%) asked for a full care plan. Gemini wrote a 6,949-character plan — DAPT duration, cardiac rehab referral, ACE inhibitor for the low EF, return precautions for chest pain. Clinically excellent. It never mentioned depression screening. Not once. GPT-5.4 Mini and Gemma 4 31B did the same.

Post-MI depression screening is guideline-recommended (AHA) — depression after MI is common and worsens outcomes. All three models wrote thorough cardiac plans with a complete psychosocial blind spot. The plans read as authoritative and complete; the omission is invisible unless you're checking for it. This is the failure mode that worries me most: not wrong answers, but confident, fluent incompleteness.

Finding 3: Gemma prescribes the drug, skips the safety precondition

Gemma 4 31B's two additional misses share a pattern worth naming:

AFib case: Gemma correctly prescribed apixaban for stroke prevention (CHA2DS2-VASc = 3) — then never assessed bleeding risk. No HAS-BLED, no bleeding-risk mention anywhere in the response, while starting an anticoagulant. The drug decision was right; the safety precondition was skipped.
Osteoporosis case: Gemma correctly started alendronate 70 mg weekly with proper administration instructions — then never mentioned dental review, despite the well-known osteonecrosis-of-the-jaw risk with bisphosphonates.

In both cases the model did the headline clinical action correctly and dropped the guardrail step. A human reviewer skimming for "did it get the treatment right?" would pass both. The checklist caught what fluency hides.

What surprised me

I expected the models to struggle most with medication safety — catching a subtle interaction feels like the hard reasoning task. It was the easiest: 24/24. The hard task turned out to be completeness under open-ended generation: when asked to "be comprehensive," every model produced something that felt comprehensive while dropping guideline-recommended items. Benchmarks that score overall quality or helpfulness would have given these plans top marks. Only checklist assertions against ground truth surfaced the gaps.

My benchmark

CareManage-Bench on Kaggle Benchmarks

24 tasks, 3 families, ground-truth checklist scoring
All patient cases fully synthetic
Reproducible: fixed prompts, published assertions, deterministic scoring

What I'd measure next

Adversarial polypharmacy: deliberately tricky multi-drug patients where the danger is a three-way interaction, not a pairwise one
A human clinician baseline: how does a practicing care manager score on the same checklist? (My guess: humans miss different items than models, and the union is the safest system)
Checklist-weighted scoring: right now a missed dental review costs the same as a missed anticoagulant. Clinically, they don't weigh the same — but deciding the weights is itself a clinical judgment worth making explicit

Takeaway

If your benchmark doesn't separately measure omissions, you're selecting for the most fluent liar. All three models I tested write beautiful care plans. Two of them are genuinely strong. But "beautiful and mostly complete" is a dangerous place to stop in healthcare — because the missing 7% is depression screening after a heart attack, bleeding risk before blood thinners, and a dental check before jaw-risk drugs. Measure the checklist, not the prose.

Built for the Kaggle x DEV Benchmarking Challenge. All 24 patient cases are synthetic — no real patient data was used.

DEV Community: Habib Ur Rehman

I Benchmarked 3 LLMs on Clinical Care-Management Tasks — They Prescribe the Drug and Skip the Safety Check

I Benchmarked 3 LLMs on Clinical Care-Management Tasks — They Prescribe the Drug and Skip the Safety Check

What I benchmarked

Models tested

Findings

The scoreboard

Finding 1: Every model aced the "scary" tasks

Finding 2: The depression-screening blind spot (all 3 models)

Finding 3: Gemma prescribes the drug, skips the safety precondition

What surprised me

My benchmark

What I'd measure next

Takeaway