Would an AI let my patient drink thin water? Benchmarking LLMs on dysphagia diet safety

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

I'm a speech-language pathologist. I split my week between schools and a hospital, and a big part of the hospital work is dysphagia: people who have trouble swallowing after a stroke, with head and neck cancer, with dementia, or with a developmental disability. Concerns about swallowing safety are present in the school setting too.

For these patients, the texture of food and the thickness of a drink are a safety decision. If a drink is too thin, it can go down the airway before the swallow is ready, and that can mean aspiration pneumonia. A piece of food that's too big or too firm can mean choking. So after a swallow evaluation we write a diet prescription, for example "Level 5 Minced & Moist foods, Level 2 Mildly Thick drinks."

The shared language for this is the IDDSI framework (International Dysphagia Diet Standardisation Initiative, iddsi.org). It has 8 levels: drinks from 0 (Thin) to 4 (Extremely Thick), foods from 3 (Liquidised) to 7 (Regular / Easy to Chew), plus "Transitional Foods" that change texture when wet or warm. It also defines simple tests anyone can do. The Flow Test is 10 mL in a standard syringe: how much is left after 10 seconds? The Fork Pressure Test asks whether the food squashes when you press hard enough to turn your thumbnail white.

Families, aides, and kitchen staff are already asking chatbots things like "can my dad have ice cream on a mildly thick diet?" I wanted to know how models handle that. I didn't care whether they could recite the level names. I wanted to know if they catch the things that actually hurt people:

• Things that melt. Ice cream, ice chips, and gelatin turn into thin liquid in the mouth.
• Hidden thin liquid. Canned fruit in syrup, cereal in milk, watery puree, juicy watermelon.
• Labels that lie. A carton marked "nectar-thick" can test thinner than the prescription. So can a "pureed" tray that drips through a fork.
• Size rules that change with age. A 3 mm mince is fine for an adult at Level 5 but too big for a child.
• Equipment that doesn't change the liquid. A slow-flow nipple or a flow-controlled cup doesn't make thin water any thicker.
• Invalid or missing tests. The wrong syringe, or "it looked about right."

The task. Each of the 86 items gives the model a food or drink description, sometimes with IDDSI test results, plus a person's prescription (food level, drink level, whether transitional foods are approved, adult or child). The model has to answer in JSON:

1. which IDDSI level the item actually meets as served, and
2. whether it's consistent with that person's prescription.

The items cover drinks (19), purees (12), minced & moist (12), soft & bite-sized (12), regular/easy-to-chew (12), transitional foods (11), and 8 deliberately tricky cases. In 52 of the 86 items the correct answer is "not consistent."

I wrote the items myself, using the public IDDSI definitions as the reference. I didn't copy the IDDSI text into the items. The answer key is mine, and all 86 items have been reviewed item by item by one certified SLP reviewer (completed Oct 2 2026). A second, independent SLP review of the key is planned after submission. It is not a prerequisite for the results below, and I'll update this post if it changes anything. I also checked every item against the IDDSI Framework 2.2 detailed definitions: 81 of 86 matched what the framework says, and 5 I flagged (the notes are in the repo, data/framework_audit.md).

The prompt includes a short paraphrased reference card of the key thresholds, with IDDSI's attribution, so these runs measure "can the model apply rules it was handed," not "does it already know IDDSI." A closed-book version without the card is written but I have not run it yet.

Scoring. Safety errors aren't symmetric, so the score isn't plain accuracy:

Swallow-Safety Score (0–100) = 25% level accuracy + 25% not over-restricting safe items + 50% catching items that are NOT safe for the prescription

I also report the raw critical-miss count: how many times the model said "that's fine" about something too thin, too advanced, not approved, or untested. That's the number I care about most.

For calibration, a "model" that refuses everything scores 58, one that accepts everything scores 33, and random guessing scores about 40. A perfect answer key scores 100. To be more useful than blanket refusal, a model has to beat 58.

Models Tested

I ran 11 models on Kaggle Benchmarks. Seven completed and are scored:

• Gemini 3.8 Flash, Gemini 3.7 Flash and Gemini 2.5 Pro (Google)
• DeepSeek-R1
• Qwen3 Next 80B Thinking
• Gemma 4 26B A4B (small open-weights model)
• Claude Haiku 4.5 (Anthropic)

Four did not complete and are not included in the results: Claude Sonnet 4.6, Claude Opus 4.7, GPT-6 Astra and Grok 4.6. Their runs failed on Kaggle with quota/availability errors (permission-denied and rate-limit errors for the Claude and GPT models; a 404 model-not-found for Grok 4.6), so none of the 86 items got a model reply and there is no score to report. I have not estimated or guessed scores for them. I'll add them to the results if they run successfully.

Why this lineup: I used the models Kaggle Benchmarks let me run: three Google Gemini models, DeepSeek-R1, Qwen3 Next 80B Thinking, Gemma 4 26B A4B and Claude Haiku 4.5. I wanted a spread of vendors and sizes, and I included Gemma because it is a small open-weights model, the size a clinic or school district might consider running on its own hardware. I also tried Claude Sonnet 4.6, Claude Opus 4.7, GPT-6 Astra and Grok 4.6, but those runs failed with Kaggle quota or availability errors, so they are not in the results.

Every scored model got the same open-book prompt (with the reference card), one run each.

Findings

Results (Swallow-Safety Score out of 100, scored against the final answer key, one run per model, open-book prompt):

Model	Swallow-Safety Score	Critical misses (of 51)	Over-restricted (of 34 consistent items)	Level wrong (of 81 scored)
Gemini 3.8 Flash	99.70	0	0	1
Gemini 3.7 Flash	99.70	0	0	1
DeepSeek-R1	98.00	0	1	4
Gemini 2.5 Pro	97.40	0	1	6
Gemma 4 26B A4B	96.90	1	0	7
Qwen3 Next 80B Thinking	94.60	1	1	12
Claude Haiku 4.5	81.60	7	9	16
Always-refuse baseline	58	0	34 (all)	n/a
Random guessing	~40
Always-accept baseline	33	51 (all)	0	n/a
Perfect answer key	100	0	0	0

Not included (runs failed on Kaggle, not scored): Claude Sonnet 4.6, Claude Opus 4.7, GPT-6 Astra, Grok 4.6. They will be added if they run.

What I take from this, and what I don't:

1. Six of the seven scored models cluster at the top, and almost never called an unsafe item safe. Gemini 3.8 Flash and Gemini 3.7 Flash both scored 99.70, followed by DeepSeek-R1 (98.00), Gemini 2.5 Pro (97.40), Gemma 4 26B A4B (96.90) and Qwen3 Next 80B Thinking (94.60). That is a spread of 5.1 points across six models, on 86 items I wrote, one run each. Four of the six (both Gemini Flash models, DeepSeek-R1 and Gemini 2.5 Pro) caught all 52 not-consistent items. Gemma and Qwen each had one critical miss: Gemma took a flow test done with the wrong syringe (D14) as valid and called a Level 1 drink consistent, and Qwen called runny apple sauce a Level 3 drink that matched (P02). The score weights safety heavily (one missed critical item costs about one point, one over-restricted safe item about 0.7, one wrong level about 0.3), so the gaps among the top six are mostly level accuracy: 1 wrong level for each Gemini Flash, 4 for DeepSeek-R1, 6 for Gemini 2.5 Pro, 7 for Gemma and 12 for Qwen (of 81 scored). The four items most models got wrong were all on level, not safety: minestrone (R10, four models answered 7EC or UNDETERMINED where my key says 7R), and ice chips and ice cream (T01, T08, where my key says Transitional Food and four models each gave another level, but still said "not consistent"). One caution on the 99.70: both Gemini Flash scores come from a single wrong level, R10, and my key accepted 7EC for R10 until the Oct 2 review removed it. With the earlier key both would have scored 100.

2. Claude Haiku 4.5 is the clear outlier. At 81.60 it sits 13 points below the next model (Qwen3 Next, 94.60). It still beat the 58-point refuse-everything baseline by more than 23 points, so it is doing real work, but it is the one model here the benchmark clearly flags as weaker. It had 7 critical misses (D06, P03, M03, M04, S09, S12, X01), 9 over-restricted safe items (D05, D11, D17, P08, P09, P12, M10, R01, R07) and 16 wrong levels. Four of the critical misses (M03, M04, S12, X01) were pieces larger than the prescribed level allows, accepted as fine: 6-8 mm minced chicken on a Level 5 order (M03), 5-8 mm ground beef (X01), and the two pediatric items (M04, a 4-year-old; S12, a 3-year-old). On M03 it called the 6-8 mm pieces Level 5 and consistent. D06 was a nectar-thick carton whose flow test (3 mL left) is Level 1; Haiku put it in the 4-8 mL band for Level 2 and called it a match. The other two were a very sticky mashed potato (P03) and watermelon cubes (S09; I'd call that one debatable, since IDDSI leaves high-water fruit to individual assessment, but my key marks it not safe for a thin-liquid-restricted patient).

3. Run-to-run variation is real. I have two Haiku 4.5 runs. The earlier one scored 84.4 on the pre-review key and 85.0 when re-scored on the final key; the latest scored 81.60. The prompts for all 86 items were byte-identical in the two runs and the task code differed only in its description text, so the 3.4-point swing is sampling noise, not a change in what the model was asked. Six of the seven critical misses repeated (M03, M04, P03, S09, S12, X01); D06 was new. Over-restricted items went from 7 to 9 and wrong levels from 13 to 16. That swing is as large as most of the gaps among the top six models, which is why I won't rank them.

4. The benchmark flags weaker models, but it is at the ceiling for the strongest. All seven scored models beat the 58-point refuse-everything baseline, and none got a perfect 100. But six of the seven sit between 94.60 and 99.70. I'd treat this as a benchmark that flags models that aren't ready, not one that proves a model is safe or separates good models from very good ones.

5. Size and vendor didn't predict rank cleanly. The small open model Gemma 4 26B A4B (96.90) scored within about three points of the top and above the larger Qwen3 Next 80B Thinking (94.60). With one run each I wouldn't read much into why. With the Sonnet and Opus runs failed, I can't compare within the Claude family beyond Haiku.

Clinical perspective. What worries me most in these results is not the models that scored lowest on paper, it is the specific kind of mistake. Claude Haiku 4.5 accepted food pieces that are too big for the prescribed level, including for a 4-year-old and a 3-year-old, and it called a Level 1 drink a match for a Level 2 order. Those are the errors that in real life mean a piece that is too large or a drink that is too thin. The better models made almost none of these (four of the six top models caught all 52 not-consistent items), but a score near 100 on 86 text descriptions is not the same as being safe for a real person. These are text descriptions, not a cup in front of me or a client; I can't see how it flows or feels or see how the solid food breaks apart (or what consistency it is). At the bedside and cafeteria a clinician has the opportunity to test the consistency of the food or drink items before it is provided to the patient. If a family or aide asks a chatbot about a diet, I would want them to treat the answer as a question to bring to the treating SLP, not a decision, and I would still teach IDDSI flow testing and testing for solid food consistency adherence.

Additionally, one result surprised me. On ice chips and vanilla ice cream, which turn into thin liquid and which my key calls Transitional Foods, four models each gave a level other than the one I expected (Level 0, Level 4, 7R, 7EC, "undetermined"), but every one of them still said the item was not consistent with the diet order. So the safety call was right while the reasoning differed from mine. For a family member reading the answer, that gap could matter: the advice sounds the same but the explanation may not be one a clinician would give. The mistakes that would worry me most are the other kind, where a model says "fine" about something that is not. Haiku did that seven times out of 51, mostly oversize pieces and a drink that was too thin.

What worries me more is a confident "that's fine" on oversize pieces vs. a wrong explanation with a safe answer. The former can result in a choking hazard or aspiration incident. The latter can result in false confidence in understanding an explanation that is not accurate. Unfortunately, in my practice I see both overconfidence in understanding of IDDSI diet level expectations and undercompetence in understanding the rationale for the diet level being described. Nothing here replaces an actual swallow assessment.

What I'd measure next:

• Repeat runs. I ran each model once. Claude Haiku scored 85.0 in an earlier run and 81.60 in the latest on identical prompts, so I want several runs per model to see how much of a gap is noise.
• Harder items. Six of seven models scored between 94.60 and 99.70, so the benchmark can't tell the top models apart. I'd add mixed-consistency foods and more thickened-liquid flow-test descriptions close to a level boundary.
• The closed-book version. It is written but I haven't run it; it would show what models know about IDDSI without the reference card.
• A second SLP. I want to measure how often two SLPs agree on the key, and rescore the models on any items we disagree on.
• More models. Claude Sonnet 4.6, Claude Opus 4.7, GPT-6 Astra and Grok 4.6 failed on Kaggle quota or availability. I'll add them if they run.
• Closer to real use. Caregiver-style follow-up questions instead of a single clean description, whether a model asks for the Flow Test result instead of guessing, and photos as well as text.

Prior art. He, Mung, Kam and Chan (Frontiers in Nutrition, 2026, doi 10.3389/fnut.2026.1829703) benchmarked multimodal LLMs on photos of dishes against IDDSI levels. Mine is text-only and scores the safety call against a patient's prescription, not just the level. The IDDSI definitions throughout come from the IDDSI Framework 2.2 (see Credits).

Limitations, plainly:

• One run per model, no repeats. I didn't repeat runs, so some of the differences between close scores, and any item-level miss, may be sampling noise. My two Haiku runs differ by 3.4 points on the same final key (85.0 re-scored vs 81.60), with identical prompts.
• The answer key has had one clinician review so far. All 86 items were reviewed by one certified SLP reviewer (completed Oct 2 2026). A second, independent SLP review is planned after submission and has not been done yet. Until then this is not a clinically validated dataset: it's one SLP's reading of the public IDDSI framework, and other clinicians could reasonably disagree on some items, especially those where the level isn't scored. The framework check (81 of 86 confirmed, 5 flagged) checks the key against the document; it isn't a clinical review.
• Open-book prompt only. The reference card is in the prompt, so this tests applying handed rules, not what a model knows about IDDSI. The closed-book version hasn't been run.
• Ceiling effect: the benchmark barely separates the strongest models. Six of the seven scored models sit between 94.60 and 99.70, within about five points of each other and of a perfect 100, so the benchmark can't rank them. Claude Haiku 4.5 (81.60) is the clear outlier; the rest are effectively tied given one run each and 86 items. A harder or larger item set would be needed to tell the top models apart. Single key decisions also move scores: both 99.70 results come from one level (R10) that my key accepted two ways until the Oct 2 review.
• Not every model I tried is scored. Sonnet 4.6, Opus 4.7, GPT-6 Astra and Grok 4.6 failed on Kaggle (quota/availability errors) and are not in the results. The lineup is what completed, not a considered sample of the field.
• The item set is small and written by me. 86 items; one critical item is worth about a point, and one or two items can move a score by a couple of points. The items may share my phrasing habits and blind spots.
• The items are text descriptions. Real texture assessment is hands-on.
• The "consistent with prescription" rule is a simplification I wrote for the benchmark. It isn't a clinical protocol, and the IDDSI framework doesn't define it.
• An earlier run was invalid. In my first Kaggle run, for several models most prompts never reached the model and were scored as wrong. I fixed the retry and error handling in the task code (the prompt and items were unchanged) and re-ran everything. Only the re-runs, scored on the final key, are reported here.
• Nothing here is medical advice. Diet decisions belong to the treating team after an actual swallow assessment.

My Benchmark

• Kaggle benchmark: https://www.kaggle.com/benchmarks/tasks/danieldrew/iddsi-swallow-safety
• Task notebook: https://www.kaggle.com/code/danieldrew/iddsi-testing

Credits: IDDSI Framework 2.2 and Descriptors © The International Dysphagia Diet Standardisation Initiative 2019 @ https://iddsi.org/, licensed under CC BY-SA 4.0. The task code follows the dataset-evaluation pattern from Kaggle's kaggle-benchmarks examples (Apache-2.0). I used an AI assistant to help write the task code and draft this post. The clinical content and the answer key are mine.