MedScribe AI — An Agentic Radiologist's Assistant

Joti Lohana — Thu, 15 Oct 2026 05:00:56 +0000

---
title: "MedScribe AI — An Agentic Radiologist's Assistant Built with Open-Source AI"
published: true

tags: devchallenge, hf26challenge, ai, healthcare

The Problem

Pakistan has approximately 1 radiologist per 100,000 people in rural areas. Radiology reporting is slow, expensive, and inaccessible — especially in low-resource healthcare settings where radiologists are scarce. Most diagnostic AI tools stop at a raw prediction, leaving clinicians unable to trust or act on the output efficiently.

MedScribe AI changes that.

What We Built

MedScribe AI is an agentic radiologist's assistant that combines computer vision with generative AI to produce full structured clinical reports from chest X-ray images — in seconds.

🔗 Live Demo: https://medscrib-ai.streamlit.app/

Built in 48 hours at the Pak Angels Generative & Agentic AI Hackathon (Cohort 11) by a team of 6 from Pakistan 🇵🇰

How It Works

The system runs in three stages:

Stage 1 — Computer Vision Model

We trained and benchmarked two models on the same held-out test set:

Model	Accuracy	Precision	Recall	F1-Score
ResNet18 (fine-tuned)	90.5%	98.4%	88.4%	93.2%
DenseNet121 TorchXRayVision	63.8%	94.8%	53.3%	68.3%

We selected ResNet18 as our primary model — it consistently outperformed the baseline on every metric, and more importantly, it caught more true pneumonia cases (higher recall), which matters most in a clinical setting.

Grad-CAM explainability was applied to visualize the exact regions of the X-ray that influenced the prediction — so clinicians can see why the model flagged something, not just what it flagged.

Stage 2 — AI Agent Pipeline (LangGraph + Groq)

The pipeline runs as a chain of four agents:

Agent	What it does
👁️ Vision	Runs the CV model, returns detected pathologies + confidence scores
🧠 Reasoning	LLM agent decides which tools to call in what order (ReAct loop)
📋 Reporting	Generates structured clinical report + plain-language patient summary
🚨 Escalation	Flags critical cases for immediate radiologist attention

The workflow is not hardcoded — the LLM decides which tools to call and in what order based on the input. This makes the system genuinely agentic, not just a scripted pipeline.

LLM used: openai/gpt-oss-120b served via Groq
Agent framework: LangGraph (ReAct agent)

Stage 3 — Application Interface (Streamlit)

Upload any chest X-ray → full clinical report in seconds
Doctor-facing report: Findings → Impression → Recommendation → Urgency → Reasoning
Patient-facing summary: Plain language, no medical jargon
Bilingual output: English + Urdu 🇵🇰 — because patients deserve to understand their own diagnosis
Downloadable PDF including original X-ray + Grad-CAM heatmap

Why Open Source AI?

openai/gpt-oss-120b via Groq — fast inference, no patient data sent to proprietary servers
TorchXRayVision — open-source medical imaging foundation model, fully auditable
LangGraph — open-source agent framework, fully customizable for clinical workflows
PyTorch + torchvision — open weights, reproducible training, no vendor lock-in

Running on open models means patient X-rays never leave the user's control — critical for healthcare privacy. In Pakistan, this also means the cost stays low enough that the product can actually work as a business.

Who Would Pay For This?

Private clinics in Pakistan and South Asia with no on-site radiologist
Telehealth platforms needing automated preliminary screening reports
Hospital chains processing high volumes of routine chest X-rays

A clinic paying $50/month for AI-powered preliminary triage is far cheaper than hiring a part-time radiologist. That's the side hustle angle — open-source AI makes the unit economics actually work.

Tech Stack

CV Model: PyTorch · ResNet18 · TorchXRayVision · Grad-CAM · scikit-learn
AI Agents: LangGraph · Groq (gpt-oss-120b) · ReAct
App: Streamlit · Python

Why Pakistan Needs This 🇵🇰

Pakistan has 1 radiologist per 100,000 people in rural areas. Most diagnostic AI stops at a prediction — MedScribe AI goes further, generating a full clinical report, flagging urgent cases, and explaining its reasoning in plain language that both doctors and patients can understand.

Open-source AI isn't just a technical choice here — for healthcare in underserved regions, it's the only choice that makes the economics work.

Limitations & What's Next

Current limitations:

Tested only on pediatric chest X-rays — performance drops on adult datasets (domain shift)
Agent output not verified by a human clinician before being shown to the user

Next steps:

Human-in-the-loop verification where radiologists can correct AI findings
Expand beyond binary classification to detect multiple pathologies
Fine-tune on multiple chest X-ray sources for better generalization

Built with open-source AI, for the patients who need it most.

DEV Community: Joti Lohana