Tabibu Update #1: Tabibu can now read your documents

John Alexander — Fri, 16 Oct 2026 11:03:46 +0000

This is the newest Tabibu update. Tabibu can now read your documents.

Tabibu is a free health-information assistant. It started in Kenya and is built for people in other countries too. Until now you could only type a question. Now, on the website, you can add a lab report, a prescription, a PDF, a Word file or a photo of one. Tabibu reads it, explains it in plain words, and gives you questions to ask your clinician.

What it accepts

PDF (text or scanned), Word (.docx), plain text and Markdown, and JPG, PNG or WEBP photos, up to 8 MB. The feature is for Basic and Pro plans. Free accounts cannot upload files.

How it works, in two steps

A vision model transcribes the file into text.
The normal text-only Tabibu pipeline explains that text.

The second step is the same one that answers typed questions, with the same safety rules. The file never goes straight to the part that talks to you.

What it will not do

It will not diagnose you.
It will not tell you a result is "normal" or "fine".
It will not tell you to start, stop or change a medicine. Doses are repeated only as printed on the document.
It does not read X-ray, CT or MRI images, or photos of skin. If you upload one, it says so. A radiology report is different: Tabibu explains it by quoting what the radiologist wrote.

I kept image reading out on purpose. Interpreting a scan or a skin photo is a medical-device question, and I would rather say "I can't" than guess.

What happens to your file

The file and the text read from it are not stored or logged. Photos have their location data (EXIF) removed. Only your question and Tabibu's answer are saved, for about 48 hours, like any chat. There is now an opt-in "Keep 30 days" button if you want to come back to a chat after you have seen a doctor. It is off unless you turn it on, and chats that involved a file cannot be kept. Answers about files never enter the shared answer cache.

How I tested it

I used made-up documents only, never real patient files: lab reports, a prescription, a Word note, radiology reports, a DICOM file. What I checked:

A prescription: doses came back exactly as printed.
An X-ray report: the answer quoted the radiologist and did not say the scan was fine.
A file with a hidden instruction telling the model to call every result normal: it ignored the instruction and said it had.
A DICOM file: it refused.

The limits, which matter more than the wins

In my testing, about 1 in 8 radiology reports ended on a safe fallback: "here is the text I read, and questions for your clinician" instead of a full explanation. It is never a wrong claim, but it is not the full answer, and I am not counting it as a success.
These were spot checks on a small set of made-up documents. I have not run a full evaluation, so I am not publishing an accuracy number.
There is no clinician review of the policy yet. Kiswahili answers are AI-reviewed only.
WhatsApp document reading is in testing and is not open to everyone.
I have not measured the cost per file end to end yet.

The limits are listed on the trust page, and I update them when they change.

One question

On a public health benchmark, a plain AI model scored higher than Tabibu on my own re-run (0.522 against 0.426, not the official leaderboard). I wrote about why here. So the test does not tell me what you want. Do you want a health AI that tells you what you have and what to take, one that explains and sends you to a clinician, or something in between? Tell me in the comments.

Tabibu gives general health information, not a diagnosis or medical advice.

We ran HealthBench on our health AI's safety layer. It scored lower than the bare model.

John Alexander — Tue, 06 Oct 2026 05:45:22 +0000

I build Tabibu, a health-information assistant. It adds a safety layer on top of a language model: emergency escalation, refusing prescription doses, answering from retrieved sources, and a reviewer that checks the output. I wanted to know what that layer costs on a public benchmark, so I ran HealthBench on it.

The short version: the full pipeline scored below the bare model it is built on, the gap shrank when I changed one policy, and measuring it surfaced a real safety bug. Here is what I did and what I think it shows.

What I ran

HealthBench is OpenAI's public benchmark of 5,000 multi-turn health conversations, each graded against rubric criteria written by physicians. Criteria add or subtract points, and a response's score is the points it earns divided by the available positive points.

I ran a 150-conversation subset, stratified by theme, leaving out the clinical note-writing theme because it is not a consumer use case. The grader is gpt-4.1-mini with a prompt I re-implemented from the paper's description. That means these numbers are not comparable to the official leaderboard. They are only useful for comparing candidates against each other on the same conversations.

Three candidates:

The full Tabibu pipeline: retrieval, input and output guardrails, the safety layer.
The bare model it runs on (gpt-6-luna in our config), with a neutral "careful health assistant" system prompt.
The bare gpt-4o-mini, which we used before.

First result

Candidate	Mean score (n=150)	95% CI
Bare gpt-6-luna	0.520	±0.046
Bare gpt-4o-mini	0.434	±0.045
Full Tabibu pipeline	0.362	±0.049

The pipeline was 0.158 below the bare model on the same conversations (paired difference, ±0.039). That is not noise.

The theme breakdown was more interesting than the headline. On emergency_referrals (16 conversations) the pipeline scored 0.553 against 0.597 for the bare model and 0.542 for gpt-4o-mini, which is within noise. On global_health (36 conversations) it scored 0.230 against 0.414.

Why a safety layer loses on this benchmark

HealthBench rewards completeness: specific doses, differential diagnoses, broad coverage. My pipeline is designed to be safer, not more complete. It declines to give numeric doses for prescription drugs, it does not diagnose, and it prefers to answer from retrieved sources instead of from the model's general knowledge. Each of those choices costs rubric points by construction.

The global_health gap had a more mundane cause. My retrieval corpus covers one country in depth, and when a question was outside it, the pipeline said it could not find the answer in its sources. At baseline that happened in 48 of 150 conversations, and those were the conversations where the bare model pulled ahead.

None of this means the bare model is safer. The benchmark does not measure that. It does mean that a team that only watches a completeness benchmark will be tempted to remove the layer that makes the product safer.

What moved the score

I changed one policy: let the assistant answer general, low-risk questions from the model's own knowledge, with a clear label saying the answer is not from its curated sources, instead of refusing. I also allowed adult amounts for a short tier of low-risk items (common nutrients, oral rehydration, everyday over-the-counter pain relievers) while keeping prescription, child and high-risk doses deferred.

Run	Mean (95% CI)
Baseline	0.383 ± 0.049
General answers, tiered doses	0.470 ± 0.048
Same, plus a "add a general-information paragraph" instruction	0.444 ± 0.050 (reverted, no gain)
Final, after a safety fix	0.426 ± 0.051
Bare model, for reference	0.522 ± 0.047

Run-to-run noise from non-deterministic generation was about ±0.03, so I would not read anything into differences smaller than that. The policy change was worth about +0.09 over baseline. The final number is lower than the intermediate one after the safety fix described in the next section.

The bug measuring found

Per-criterion results let me read the failures instead of the averages. In doing that I found that, with the original rules, the pipeline could quote prescription doses straight out of a retrieved national formulary: a morphine dosing interval, a once-weekly methotrexate dose, a lithium range, a child's amoxicillin dose. My dose rule needed a known drug name near the number plus an instruction verb, and these passages did not match.

I replaced it with a rule on amount plus dosing schedule, added regression tests built from the real leaked answers, and re-ran. The score went from 0.470 to 0.426, which fits with some of those answers having earned points, although the difference is close to the run-to-run noise so I would not lean on it. Either way I think it is the right trade, and it is also the main reason I distrust any health-AI score reported without the safety behaviour attached to it.

Two smaller fixes came out of the same reading. A research question that merely contained the word "stroke" was triggering the emergency card, so I added must-not-escalate cases next to the must-escalate ones (all earlier cases still pass). And my output reviewer could not see earlier turns, so it judged a legitimate follow-up as "assuming a symptom never mentioned" and replied with a canned off-topic message. That scored zero.

What this does not show

It is one run per candidate on 150 conversations, with ±0.03 generation noise.
The grader is a smaller model with a prompt I re-wrote. It is not the official one, and there is no human check.
The final run was measured before one last small fix to the dose rule and was not re-run.
Most HealthBench conversations are general or US-centred, and my corpus is narrow, so the pipeline is partly being scored on scope.
A score is not a safety certification. I have no clinician review of the safety rules yet.
At roughly US$6.60 of API spend, this is a cheap experiment, not a definitive evaluation.

What I would take from it

If you build on top of a model, run the benchmark with and without your safety layer on the same conversations and publish both numbers. Read the per-criterion failures, not just the mean, because that is where the real bugs were. And be suspicious of a health assistant that reports only a benchmark that rewards saying more.

I am the builder of Tabibu, and its limits and test results are listed at tabibu.health/trust. I would like to hear how you are measuring safety in your own health-AI work.

DEV Community: John Alexander