In most jobs, when an AI is wrong, you fix the draft. In health, you might not get the chance.
I'm John, a senior AI and full-stack engineer. I spent the last year building a health-information assistant, Tabibu, and I'm starting this series with the question I get asked most: why is health so much harder than the other places I have shipped AI?
1. A mistake costs more
A wrong dose is not a typo. In a normal product, a bad answer costs a retry. In health, someone may act on it. So I stopped asking "is the answer good?" and started asking "what is the worst believable wrong answer here?" For Tabibu the answer was a dose, so the system does not tell anyone to start, stop or change a medicine.
2. The AI sounds equally sure whether it is right or wrong
This is the one that surprised me as an engineer. A model gives a calm, fluent answer when it knows and when it is guessing. A patient usually cannot tell the difference, because telling the difference is the expert's job.
So I stopped trusting tone as a signal. Answers from Tabibu show their sources, and when the guidelines do not cover a question it says so and labels the answer as general information. A confident voice is not a reason to trust anything.
3. The rules are strict for good reasons
Where software influences a medical decision, regulators in many places treat it as a medical device, and that takes evidence. I do not make that claim for Tabibu: it is an information tool, it has no clinician review yet, and the limits are listed at tabibu.health/trust.
For a builder, the lesson is to design the product so its boundaries are written down and testable, not implied.
Is it hopeless?
No. A 2026 Mount Sinai study in Communications Medicine tested 20 AI models on more than 10 million responses. Adding a short safety reminder cut potentially harmful clinical choices from 16.6% to 10.1%, with reductions in 19 of the 20 models. I read the MedicalXpress write-up, not the paper itself. Its lead researcher said a reminder is one safeguard, not a substitute for clinical oversight. That is the shape I trust: safety work helps, and it does not reach zero.
ECRI's 2026 patient safety report also flags AI in diagnostic care as a concern, which fits what I have seen: the question is not whether AI is good or bad for health, but where it is safe enough, and who checks.
What I'd ask you
What is one health decision you would never leave to an AI? Over the next few weeks I'll cover who builds these tools, how they make money, who sets the rules, and where it fails.
I post short videos in this series on LinkedIn and TikTok; come say hi.
General information, not medical advice.
Top comments (0)