---
title: "MedScribe AI — An Agentic Radiologist's Assistant Built with Open-Source AI"
published: true
tags: devchallenge, hf26challenge, ai, healthcare
The Problem
Pakistan has approximately 1 radiologist per 100,000 people in rural areas. Radiology reporting is slow, expensive, and inaccessible — especially in low-resource healthcare settings where radiologists are scarce. Most diagnostic AI tools stop at a raw prediction, leaving clinicians unable to trust or act on the output efficiently.
MedScribe AI changes that.
What We Built
MedScribe AI is an agentic radiologist's assistant that combines computer vision with generative AI to produce full structured clinical reports from chest X-ray images — in seconds.
🔗 Live Demo: https://medscrib-ai.streamlit.app/
Built in 48 hours at the Pak Angels Generative & Agentic AI Hackathon (Cohort 11) by a team of 6 from Pakistan 🇵🇰
How It Works
The system runs in three stages:
Stage 1 — Computer Vision Model
We trained and benchmarked two models on the same held-out test set:
| Model | Accuracy | Precision | Recall | F1-Score |
|---|---|---|---|---|
| ResNet18 (fine-tuned) | 90.5% | 98.4% | 88.4% | 93.2% |
| DenseNet121 TorchXRayVision | 63.8% | 94.8% | 53.3% | 68.3% |
We selected ResNet18 as our primary model — it consistently outperformed the baseline on every metric, and more importantly, it caught more true pneumonia cases (higher recall), which matters most in a clinical setting.
Grad-CAM explainability was applied to visualize the exact regions of the X-ray that influenced the prediction — so clinicians can see why the model flagged something, not just what it flagged.
Stage 2 — AI Agent Pipeline (LangGraph + Groq)
The pipeline runs as a chain of four agents:
| Agent | What it does |
|---|---|
| 👁️ Vision | Runs the CV model, returns detected pathologies + confidence scores |
| 🧠 Reasoning | LLM agent decides which tools to call in what order (ReAct loop) |
| 📋 Reporting | Generates structured clinical report + plain-language patient summary |
| 🚨 Escalation | Flags critical cases for immediate radiologist attention |
The workflow is not hardcoded — the LLM decides which tools to call and in what order based on the input. This makes the system genuinely agentic, not just a scripted pipeline.
LLM used: openai/gpt-oss-120b served via Groq
Agent framework: LangGraph (ReAct agent)
Stage 3 — Application Interface (Streamlit)
- Upload any chest X-ray → full clinical report in seconds
- Doctor-facing report: Findings → Impression → Recommendation → Urgency → Reasoning
- Patient-facing summary: Plain language, no medical jargon
- Bilingual output: English + Urdu 🇵🇰 — because patients deserve to understand their own diagnosis
- Downloadable PDF including original X-ray + Grad-CAM heatmap
Why Open Source AI?
- openai/gpt-oss-120b via Groq — fast inference, no patient data sent to proprietary servers
- TorchXRayVision — open-source medical imaging foundation model, fully auditable
- LangGraph — open-source agent framework, fully customizable for clinical workflows
- PyTorch + torchvision — open weights, reproducible training, no vendor lock-in
Running on open models means patient X-rays never leave the user's control — critical for healthcare privacy. In Pakistan, this also means the cost stays low enough that the product can actually work as a business.
Who Would Pay For This?
- Private clinics in Pakistan and South Asia with no on-site radiologist
- Telehealth platforms needing automated preliminary screening reports
- Hospital chains processing high volumes of routine chest X-rays
A clinic paying $50/month for AI-powered preliminary triage is far cheaper than hiring a part-time radiologist. That's the side hustle angle — open-source AI makes the unit economics actually work.
Tech Stack
CV Model: PyTorch · ResNet18 · TorchXRayVision · Grad-CAM · scikit-learn
AI Agents: LangGraph · Groq (gpt-oss-120b) · ReAct
App: Streamlit · Python
Why Pakistan Needs This 🇵🇰
Pakistan has 1 radiologist per 100,000 people in rural areas. Most diagnostic AI stops at a prediction — MedScribe AI goes further, generating a full clinical report, flagging urgent cases, and explaining its reasoning in plain language that both doctors and patients can understand.
Open-source AI isn't just a technical choice here — for healthcare in underserved regions, it's the only choice that makes the economics work.
Limitations & What's Next
Current limitations:
- Tested only on pediatric chest X-rays — performance drops on adult datasets (domain shift)
- Agent output not verified by a human clinician before being shown to the user
Next steps:
- Human-in-the-loop verification where radiologists can correct AI findings
- Expand beyond binary classification to detect multiple pathologies
- Fine-tune on multiple chest X-ray sources for better generalization
Built with open-source AI, for the patients who need it most.
Top comments (0)