MedScribe AI — An Agentic Radiologist's Assistant

Joti Lohana — Thu, 15 Oct 2026 05:00:56 +0000

---
title: "MedScribe AI — An Agentic Radiologist's Assistant Built with Open-Source AI"
published: true

tags: devchallenge, hf26challenge, ai, healthcare

The Problem

Pakistan has approximately 1 radiologist per 100,000 people in rural areas. A patient in Nawabshah, Sukkur, or Larkana may wait days — sometimes weeks — for a chest X-ray report that a radiologist in Karachi hasn't had time to read yet.

The tools that exist today make this worse, not better. Most diagnostic AI stops at a raw prediction — "Pneumonia: 87% confidence" — with no explanation, no report, no next step. A doctor in a rural clinic cannot act on that. They need a structured report, a recommended course of action, and ideally something their patient can actually understand.

MedScribe AI was built to close that gap.

What We Built

MedScribe AI is an agentic radiologist's assistant. Upload a chest X-ray, and within seconds you get:

A full structured clinical report — Findings, Impression, Recommendation, Urgency, Reasoning
A plain-language patient summary in English or Urdu 🇵🇰
A Grad-CAM heatmap showing exactly which regions of the X-ray influenced the diagnosis
A downloadable PDF combining all of the above

🔗 Live Demo: https://medscrib-ai.streamlit.app/

The Real Problem With AI in Healthcare

Most AI tools for radiology are black boxes. They give you a number. They don't tell you why.

A clinician who cannot explain why the AI flagged something cannot defend that decision to a patient, a colleague, or a regulator. So they ignore the AI and go back to waiting for a radiologist.

We built explainability in from the start — not as an afterthought. Grad-CAM highlights the exact lung regions that drove the prediction. The reasoning agent explains its confidence and flags uncertainty when it exists. The escalation agent triggers an alert when the case is critical, so a human radiologist reviews it first.

How It Works

Stage 1 — Computer Vision Model

We trained and benchmarked two models on the same held-out test set:

Model	Accuracy	Precision	Recall	F1-Score
ResNet18 (fine-tuned)	90.5%	98.4%	88.4%	93.2%
DenseNet121 TorchXRayVision (zero-shot)	63.8%	94.8%	53.3%	68.3%

We chose ResNet18 — not because it had the highest precision, but because it had the highest recall. In a clinical setting, missing a true pneumonia case (false negative) is far more dangerous than a false alarm. Recall is the number that matters.

Grad-CAM was applied to visualize which regions of the X-ray most influenced each prediction — making the model's reasoning visible to the clinician.

Stage 2 — AI Agent Pipeline

The pipeline runs as a chain of four agents using LangGraph and Groq:

Agent	What it does
👁️ Vision	Runs CV model, returns detected pathologies + confidence scores
🧠 Reasoning	LLM agent decides which tools to call (ReAct loop)
📋 Reporting	Generates structured clinical report + plain-language patient summary
🚨 Escalation	Flags critical cases for immediate radiologist attention

The workflow is not hardcoded. The LLM decides which tools to call and in what order based on the input — this is a genuinely agentic system, not a scripted pipeline dressed up as one.

LLM: openai/gpt-oss-120b via Groq
Framework: LangGraph (ReAct agent)

Stage 3 — Application Interface

Built with Streamlit:

Chest X-ray upload with language selector (English / Urdu)
Clearly separated doctor-facing and patient-facing sections
Downloadable PDF report including original X-ray + Grad-CAM heatmap

Why Open Source AI?

Every component of MedScribe AI is open source:

TorchXRayVision — open-source medical imaging foundation model
openai/gpt-oss-120b via Groq — open-weight LLM, fast inference
LangGraph — open-source agent framework, fully auditable
PyTorch + ResNet18 — open weights, reproducible training

This matters for three reasons:

1. Privacy — Patient X-rays never leave the clinic's control. No data is sent to a proprietary server. In healthcare, this is not optional.

2. Cost — A rural clinic in Sindh cannot pay $500/month for a closed AI API. Open models make the unit economics work. A clinic paying $30–50/month for preliminary AI triage is viable. That same service built on proprietary APIs is not.

3. Trust — Open weights mean the model can be audited, fine-tuned, and improved. A black-box API cannot be corrected when it's wrong. An open model can.

Who Would Pay For This?

Private clinics in rural Pakistan and South Asia with no on-site radiologist — they need a first-pass screening tool that flags critical cases before a radiologist opens the scan
Telehealth platforms needing automated preliminary reports at scale
Hospital chains processing high volumes of routine chest X-rays

A clinic paying $50/month for AI-powered preliminary triage is far cheaper than flying in a radiologist. That's the side hustle: open-source AI makes the economics work where closed AI doesn't.

Tech Stack

CV Model: PyTorch · ResNet18 · TorchXRayVision · Grad-CAM · scikit-learn
AI Agents: LangGraph · Groq (openai/gpt-oss-120b) · ReAct
App: Streamlit · Python

Limitations — We Are Honest About These

Tested on pediatric chest X-rays — performance drops on adult datasets (domain shift confirmed: accuracy fell from 90.5% to 65.7% on an external adult dataset)
Agent output is not verified by a human clinician before being shown — this is a screening tool, not a replacement for a radiologist
English and Urdu only for now

What's Next

Human-in-the-loop verification — radiologists correct AI findings, corrections feed back to fine-tune the model
Expand beyond binary classification to detect multiple pathologies
Domain adaptation for adult chest X-rays
WhatsApp integration — so rural clinics can upload X-rays without a laptop

Why This Matters

1 radiologist per 100,000 people. That is the reality in rural Pakistan.

MedScribe AI does not replace radiologists. It makes sure the cases that need a radiologist most — the critical ones — get to one faster. And it makes sure the patients who will never see a radiologist at all get something: an explanation, in their own language, of what the AI found.

Open source AI is not just a technical choice here. It is the only choice that makes this work.

Built by a team of 6 from Pakistan 🇵🇰

DEV Community: Joti Lohana