I was ordering medicines online late one night, staring at a scrawled prescription slip, when a random thought hit me: Why can't I just upload this image and have the exact medicines with their right dosages automatically added to my cart?
Initially, I set out to build a grand "self-learning, self-improving" ML pipeline. Fast forward a few months: folders upon folders of complex code, zero visible results, and even AI "vibe coding" couldn't bail me out. During a casual chat, a friend gave me a reality check—I was overengineering a massive enterprise storage and continuous-learning system without the data or infrastructure to support it.
That talk refocused my scope completely. I pivoted from an overambitious ML pipe-dream to Prescription Analyzer AI: a lightweight, fast tool designed to take a prescription image, process it in seconds, and spit back a clean, structured medical breakdown.
About the Project:
Prescription Analyzer AI is an open-source, bilingual medical prescription analyzer. It extracts patient demographics, doctor metadata, diagnostic notes, and structured medication tables from both handwritten and printed prescriptions.
Tech Stack Architecture
Tech Stack:
| Layer | Technologies |
|---|---|
| Frontend | React 18, GSAP, Framer Motion, Tailwind CSS |
| Backend | Go (Gin), MongoDB Driver |
| ML Engine | Python + FastAPI, OpenCV, PyTesseract |
| Models | Gemini 2.0 Flash Vision (primary), Cohere Command‑R Plus (secondary) |
| Infra | Docker, Railway, Vercel |
One detail I'm particularly proud of handling is regional Hindi medical shorthand. Parsing symbols like bd (twice daily), od (once daily), TDS, SOS, and dot-line frequency marks is what separates a real-world utility from a toy project.
Picture to abhi baaki hai, mere dost... 🎬
To be completely honest, the system is far from flawless. Here are the three biggest hurdles I'm still battling:
- Language Switching Glitches: While the pipeline detects languages reliably, asking the system to translate a Hindi prescription into a structured English output still causes occasional formatting hiccups.
- The "95% Accuracy" Trap: A 95% accuracy metric sounds great on paper until you realize the model hallucinated or swapped two critical letters in a brand-name drug. In healthcare, 5% error is huge.
- Severe Medical Handwriting: Traditional OCR requires clean, separated characters. When a doctor's handwriting degenerates into continuous wavy lines—the kind even human pharmacists pause to squint at—the baseline OCR engine completely collapses.
Doctor’s Handwriting vs. Standard OCR
The Root Problem
Standard OCR engines like Tesseract and EasyOCR are fundamentally trained on fonts: crisp glyphs, distinct character boundaries, and predictable spacing. Doctor's handwriting breaks every single one of those assumptions with connected cursive strokes, personalised shorthand, symbols, and irregular line slants.
The First Failure
When I ran my initial handwritten test sample through Tesseract, the output was brutal: 26% confidence score, 0 medicines extracted, and empty patient fields. My first instinct was to blame my backend pipeline or image-parsing code, before realising I was fighting a fundamental tool limitation.
What I Tried (And What Actually Worked)
| What I Tried | What Happened | Reference |
|---|---|---|
| Reduced 20 OCR passes to 2 smart passes (PSM 6 + PSM 11) | Resolved execution timeouts, but accuracy remained terrible. | Issue #1 |
| Upscaled image cap (1200px → 2000px), added denoising + sharpening before CLAHE | Preserved fine ink strokes; confidence bumped up slightly. | Poor OCR Quality section |
| Added PSM 11 fallback when PSM 6 confidence dropped below 0.40 | Minor improvement, but still hard-capped by Tesseract's font-based design. | Poor OCR Quality section |
| Pivoted primary engine to Gemini 2.0 Flash Vision | The actual unlock: Vision-LLMs read handwriting contextually rather than character-by-character. | Issue #5 |
Switching to a Vision-LLM shifted the paradigm from character recognition to contextual reading. It behaves much closer to a human pharmacist who infers an ambiguous word based on dosage context (500mg) rather than isolated letterforms.
Decoding the Shorthand
As a developer without a medical degree, decoding shorthand required a lot of research and analyzing failure logs:
-
bd➔ Twice daily -
×2(circled) ➔ 2 weeks duration -
c/o➔ Complaints of (maps directly to the Diagnosis field)
Results Comparison
| Metric | Tesseract Baseline | Gemini 2.0 Flash Vision |
|---|---|---|
| Confidence Score | 26% | 95–99% |
| Medicines Extracted | 0 / 4 | 4 / 4 |
| Primary Engine | Tesseract | Gemini 2.0 Flash Vision |
The Open Debate
- Is a general Vision-LLM the long-term solution, or just a temporary fix until we train lightweight, handwriting-specific medical HTR models?
- Where is the safety line? At what point is an automated extractor "good enough" to assist a pharmacist without human-in-the-loop validation?
- Privacy vs. Capability: Cloud vision APIs handle complex cursive exceptionally well, but offloading sensitive medical documents to cloud endpoints carries privacy trade-offs.
Personally, I feel Vision-LLMs are the right bridge for now, but local/on-device privacy-first models are the ultimate end goal for healthtech.
4. Open Invitation & Next Steps
I’m actively looking for contributors, testers, and feedback to make this pipeline faster and safer!
Good First Issues & Roadmap
- 🔊 Text-to-Speech: Adding voice readouts for dosage instructions.
- 🎨 Accessibility: High-contrast / color-blind UI toggles.
- 🌐 Expanded Language Support: Extending regional script parsing from Hindi to Bengali and Tamil.
- ⚠️ Drug Interaction Safety Checks: Cross-referencing extracted medications against basic interaction APIs.
How You Can Help:
- Test it out: Drop real (anonymized!) prescription edge-cases into our issues tab.
- Code & Architecture: If you've solved handwriting OCR or non-Latin script parsing, I'd love to hear your approach!
- Server Hosting Suggestions: The live demo currently runs on free-tier Render instances (which spin down after inactivity). If you know better hosting platforms for small open-source backends, drop a suggestion in the comments!
🔗 GitHub Repository: Snehadas2005/Prescription-Analyzer-AI
Top comments (0)