My Aaji Can't Read Her Prescription. So I Built Her a Companion.

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

My Aaji is comfortable in Marathi.

A prescription, unfortunately, doesn't care.

It comes home as a small piece of paper filled with medicine names, abbreviations like 1-0-1, and instructions like "after food" and somehow, she's expected to figure it all out and remember it every single day.

I kept thinking: we have AI that can write essays, generate code and hold a conversation. Why can't I make something that simply makes this one thing easier for her?

So I built Med Companion.

What I Built

Med Companion turns a photo of a prescription into verified medication reminders in her own language.

• She (or a family member) takes a photo.
• Gemma extracts what is actually written.
• A family member checks it against the original prescription.
• Temporal makes sure a reminder doesn't disappear if nobody responds.
• When it's time, the reminder is read aloud in Marathi, Hindi or Tamil.

She can simply say:

"मी घेतली."
I took it.

And if a dose still isn't confirmed after a couple of reminders, it shows up in a Family view, so someone can check in with a phone call instead of finding out days later.

There is one rule behind all of it:

The AI is never allowed to guess.

If the prescription doesn't print a time, Med Companion doesn't invent one.
If a medicine name isn't clear, it asks for verification.
Nothing gets scheduled until a person checks the extracted details against the original.

Because with medicine, "I'm not sure" is a much better answer than a confident wrong one.

Try it in 30 seconds: open the app, tap Try sample prescription (it's fictional), watch it stop at "Do not schedule yet" until every detail is verified, then confirm and hear the Marathi reminder.

Demo

Untitled design (1).mp4 - Google Drive

drive.google.com

The demo uses a fictional, clearly labelled sample prescription and walks through the whole journey:

photo → extraction → caregiver verification → reminder → voice confirmation → family view

(Demo mode compresses reminder times into seconds, so you don't have to wait until 9 AM.)

Code

TruptiLande / med-companion

Med Companion

Prescription photo -> PaddleOCR -> Gemma (Ollama) -> caregiver confirms -> Temporal reminders -> ElevenLabs voice -> family alert The AI only explains what is printed on the prescription. It never prescribes or changes doses.

Run

1. ollama pull gemma3:4b (or set GEMMA_MODEL to any Gemma tag you have)
2. py -3.12 -m pip install -r backend/ocr/requirements.txt (EasyOCR models download the first time fallback is needed)
3. temporal server start-dev
4. cd backend && cp .env.example .env && npm i && npm run worker and, in a second terminal, npm run dev
5. cd frontend && npm i && npm run dev -> http://localhost:5173 Set DEMO=1 in .env so reminders fire in seconds (retry after 60s) instead of at clock time. Set OCR_CONFIDENCE_THRESHOLD in backend/.env to adjust when EasyOCR fallback is attempted. Low-confidence results stay in caregiver review.

Voice reminders

Automatic spoken reminders are enabled while the app is open. If your browser…

View on GitHub

How I Built It

Prescription photo
     ↓
PaddleOCR / EasyOCR
     ↓
Gemma (local, via Ollama)
     ↓
Human verification
     ↓
Temporal → reminders → family view
     ↓
Marathi / Hindi / Tamil voice

• Gemma + Ollama extract the medicine name, strength and frequency at temperature: 0, with the output validated by a strict schema. Anything unclear comes back as null and goes to review instead of being guessed.
• PaddleOCR / EasyOCR read the photo locally. When confidence is low, the app says so and keeps the prescription in review.
• Temporal runs every dose as a durable workflow: wait, remind, listen for "taken" or "snooze", retry, then mark it missed and flag the family. It survives restarts, which matters for something people rely on.
• Mastra powers a small assistant that answers questions like "What medicines do I take today?" using only the confirmed schedule. It refuses to give medical advice.
• ElevenLabs provides the spoken Marathi, Hindi and Tamil. It's optional: without it, the app falls back to browser speech or plain text.
• React + Vite for the interface, local MongoDB for state.

Why Does Open Innovation Matter?

A prescription contains someone's name, medicines and health information. For something this personal, I wanted control and privacy to be part of the design, not an afterthought.

Running the core AI locally with open pieces gave me things a closed API couldn't:

• Her prescription doesn't leave the machine just to be understood. OCR, Gemma, Temporal and the database all run locally. Only the optional voice feature needs the internet.
• Control over behavior, not just prompts. Because I run the model myself, I could enforce strict schemas, null over guesses, and a mandatory human check.
• No per-reminder cost. A reminder app that nudges someone daily for years shouldn't come with a metered bill.
• It's swappable. Changing the model is a single setting, so a better open Marathi-capable model can drop in later.

ElevenLabs is a closed service, so I kept it optional on purpose. The safety loop never depends on it.

Prize Categories

• Best Use of Gemma: local Gemma for strict, validated extraction and plain-language explanations.
• Best Use of Temporal: a durable workflow for every dose, with snooze, retries, missed-dose handling and family escalation.
• Best Use of Mastra: a tool-using schedule assistant that answers only from the confirmed schedule.
• Best Use of ElevenLabs: spoken multilingual reminders, with graceful fallback when offline.

Why I Built It

This isn't meant to replace a doctor.
It isn't meant to replace a pharmacist.
And it definitely isn't meant to replace family.

It's just meant to make one small part of taking medicine a little less confusing for someone I love.

I'm still waiting to properly hand it over to my Aaji, so I won't invent a reaction for the sake of this post.

But I know what I hope happens.

The next time she looks at that little prescription slip, she won't have to figure it out alone.

काळजी घ्या, Aaji. ("Take care.")

---

Med Companion only uses information explicitly present on a prescription. It never prescribes, changes doses, or provides medical advice. Every schedule must be verified by a person before reminders begin.