---
title: "MedScribe AI — An Agentic Radiologist's Assistant Built with Open-Source AI"
published: true
tags: devchallenge, hf26challenge, ai, healthcare
The Problem
Pakistan has approximately 1 radiologist per 100,000 people in rural areas. A patient in Nawabshah, Sukkur, or Larkana may wait days — sometimes weeks — for a chest X-ray report that a radiologist in Karachi hasn't had time to read yet.
The tools that exist today make this worse, not better. Most diagnostic AI stops at a raw prediction — "Pneumonia: 87% confidence" — with no explanation, no report, no next step. A doctor in a rural clinic cannot act on that. They need a structured report, a recommended course of action, and ideally something their patient can actually understand.
MedScribe AI was built to close that gap.
What We Built
MedScribe AI is an agentic radiologist's assistant. Upload a chest X-ray, and within seconds you get:
- A full structured clinical report — Findings, Impression, Recommendation, Urgency, Reasoning
- A plain-language patient summary in English or Urdu 🇵🇰
- A Grad-CAM heatmap showing exactly which regions of the X-ray influenced the diagnosis
- A downloadable PDF combining all of the above
🔗 Live Demo: https://medscrib-ai.streamlit.app/
The Real Problem With AI in Healthcare
Most AI tools for radiology are black boxes. They give you a number. They don't tell you why.
A clinician who cannot explain why the AI flagged something cannot defend that decision to a patient, a colleague, or a regulator. So they ignore the AI and go back to waiting for a radiologist.
We built explainability in from the start — not as an afterthought. Grad-CAM highlights the exact lung regions that drove the prediction. The reasoning agent explains its confidence and flags uncertainty when it exists. The escalation agent triggers an alert when the case is critical, so a human radiologist reviews it first.
How It Works
Stage 1 — Computer Vision Model
We trained and benchmarked two models on the same held-out test set:
| Model | Accuracy | Precision | Recall | F1-Score |
|---|---|---|---|---|
| ResNet18 (fine-tuned) | 90.5% | 98.4% | 88.4% | 93.2% |
| DenseNet121 TorchXRayVision (zero-shot) | 63.8% | 94.8% | 53.3% | 68.3% |
We chose ResNet18 — not because it had the highest precision, but because it had the highest recall. In a clinical setting, missing a true pneumonia case (false negative) is far more dangerous than a false alarm. Recall is the number that matters.
Grad-CAM was applied to visualize which regions of the X-ray most influenced each prediction — making the model's reasoning visible to the clinician.
Stage 2 — AI Agent Pipeline
The pipeline runs as a chain of four agents using LangGraph and Groq:
| Agent | What it does |
|---|---|
| 👁️ Vision | Runs CV model, returns detected pathologies + confidence scores |
| 🧠 Reasoning | LLM agent decides which tools to call (ReAct loop) |
| 📋 Reporting | Generates structured clinical report + plain-language patient summary |
| 🚨 Escalation | Flags critical cases for immediate radiologist attention |
The workflow is not hardcoded. The LLM decides which tools to call and in what order based on the input — this is a genuinely agentic system, not a scripted pipeline dressed up as one.
LLM: openai/gpt-oss-120b via Groq
Framework: LangGraph (ReAct agent)
Stage 3 — Application Interface
Built with Streamlit:
- Chest X-ray upload with language selector (English / Urdu)
- Clearly separated doctor-facing and patient-facing sections
- Downloadable PDF report including original X-ray + Grad-CAM heatmap
Why Open Source AI?
Every component of MedScribe AI is open source:
- TorchXRayVision — open-source medical imaging foundation model
- openai/gpt-oss-120b via Groq — open-weight LLM, fast inference
- LangGraph — open-source agent framework, fully auditable
- PyTorch + ResNet18 — open weights, reproducible training
This matters for three reasons:
1. Privacy — Patient X-rays never leave the clinic's control. No data is sent to a proprietary server. In healthcare, this is not optional.
2. Cost — A rural clinic in Sindh cannot pay $500/month for a closed AI API. Open models make the unit economics work. A clinic paying $30–50/month for preliminary AI triage is viable. That same service built on proprietary APIs is not.
3. Trust — Open weights mean the model can be audited, fine-tuned, and improved. A black-box API cannot be corrected when it's wrong. An open model can.
Who Would Pay For This?
- Private clinics in rural Pakistan and South Asia with no on-site radiologist — they need a first-pass screening tool that flags critical cases before a radiologist opens the scan
- Telehealth platforms needing automated preliminary reports at scale
- Hospital chains processing high volumes of routine chest X-rays
A clinic paying $50/month for AI-powered preliminary triage is far cheaper than flying in a radiologist. That's the side hustle: open-source AI makes the economics work where closed AI doesn't.
Tech Stack
CV Model: PyTorch · ResNet18 · TorchXRayVision · Grad-CAM · scikit-learn
AI Agents: LangGraph · Groq (openai/gpt-oss-120b) · ReAct
App: Streamlit · Python
Limitations — We Are Honest About These
- Tested on pediatric chest X-rays — performance drops on adult datasets (domain shift confirmed: accuracy fell from 90.5% to 65.7% on an external adult dataset)
- Agent output is not verified by a human clinician before being shown — this is a screening tool, not a replacement for a radiologist
- English and Urdu only for now
What's Next
- Human-in-the-loop verification — radiologists correct AI findings, corrections feed back to fine-tune the model
- Expand beyond binary classification to detect multiple pathologies
- Domain adaptation for adult chest X-rays
- WhatsApp integration — so rural clinics can upload X-rays without a laptop
Why This Matters
1 radiologist per 100,000 people. That is the reality in rural Pakistan.
MedScribe AI does not replace radiologists. It makes sure the cases that need a radiologist most — the critical ones — get to one faster. And it makes sure the patients who will never see a radiologist at all get something: an explanation, in their own language, of what the AI found.
Open source AI is not just a technical choice here. It is the only choice that makes this work.
Built by a team of 6 from Pakistan 🇵🇰
Top comments (0)