DoseVoice: I built an AI that reads pill bottles out loud (and knows when to say 'turn the bottle')

haricharanparna — Sun, 15 Nov 2026 04:02:20 +0000

This is a submission for the MLH x DEV Writing Challenge

Built in 36 hours at HackUMass XIV.
Try it: dosevoice.vercel.app · Code: github.com/haricharanparna/dosevoice
Demo Video https://youtu.be/KVYxPievtLg

The Problem

I built DoseVoice because of my grandma. She was often squinting at her medication bottles and containers, struggling to read the labels, and she often didn't know how much of a medicine to take. That's very dangerous: it can lead to an overdose, or to taking far too little for the medicine to work.

Prescription labels are some of the smallest, most important text people read. The people who take the most medications, older adults and people with low vision, also have the hardest time reading them. A misread label or a forgotten dose can mean a hospital visit.

So we built DoseVoice: point your camera at a pill bottle, hear exactly what it says, and get spoken reminders so you never miss a dose.

What it does

Scan: a live camera opens right in the browser. Gemini reads the label, and DoseVoice speaks it in plain language ("500 milligrams", not "500 mg").
"Turn the bottle": if the photo only shows the name, DoseVoice says so out loud and asks for the directions side, then reads both photos together.
Reminders that talk: reminder times come straight from the label ("twice daily" becomes 9 AM and 9 PM). At dose time it speaks the reminder and sends a notification with I took it and Snooze buttons.
Safety check: Gemini looks across all your medications for documented interactions. In our demo it flagged warfarin + ibuprofen as a major bleeding risk and turned it into a question for your pharmacist.
Built for low vision: three text sizes, high contrast, dark mode, adjustable speaking speed, and spoken feedback for every action.

The stack

Piece	What we used
Frontend	Next.js 15 (App Router), React 19, TypeScript, a custom CSS design system
AI	Google Gemini API (multimodal vision + structured output)
Database	MongoDB Atlas (free M0 cluster)
Browser APIs	`getUserMedia` (camera), Web Speech API (voice), service worker (notifications)
Hosting	Vercel (free Hobby plan)
AI pair programmer	Claude (Anthropic) helped debug and refine the code; I directed the features, made the code, and tested/made the product decisions

Total cost: $0. Everything runs on free tiers.

The interesting part: making an LLM trustworthy enough for medication

Free-text answers from an LLM are a non-starter when the subject is dosing. So every Gemini call uses a strict JSON response schema:

const SCHEMA = {
  type: "OBJECT",
  properties: {
    isMedicationLabel: { type: "BOOLEAN" },
    drugName: { type: "STRING" },
    strength: { type: "STRING" },
    instructions: { type: "STRING" },
    suggestedTimes: { type: "ARRAY", items: { type: "STRING" } },
    warnings: { type: "ARRAY", items: { type: "STRING" } },
    plainSummary: { type: "STRING" },
    missing: { type: "ARRAY", items: { type: "STRING", enum: ["name", "strength", "directions"] } },
    retakeHint: { type: "STRING" },
    confidence: { type: "STRING", enum: ["high", "medium", "low"] },
  },
  required: [/* all of the above */],
};

Two fields do the heavy lifting:

confidence: when it's low, DoseVoice says so out loud and tells you to confirm with your pharmacist instead of pretending.
missing: this is what powers the "turn the bottle" feature. Pill bottles are round, so one photo usually shows half the label. Rather than guessing directions, Gemini reports what it couldn't see, and the app asks for another photo.

I also don't fully trust the model to flag its own gaps, so the server double-checks:

// If Gemini left directions blank but didn't flag it, flag it ourselves.
if (scan.isMedicationLabel && !scan.instructions?.trim() && !scan.missing.includes("directions")) {
  scan.missing.push("directions");
}

Reading two sides of a bottle at once

Gemini accepts multiple images in one request, so "turn the bottle" is just one more photo appended to the same call:

const parts = images.map((img) => {
  const [, mimeType, data] = img.match(/^data:(image\/[\w+.-]+);base64,(.+)$/)!;
  return { inline_data: { mime_type: mimeType, data } };
});
await geminiJSON<Scan>([...parts, { text: PROMPT }], SCHEMA);

The prompt tells Gemini the photos may be different sides of the same bottle and to combine them, so the second scan returns one complete result (name from photo 1, directions from photo 2).

Things that broke (and how I fixed them)

1. The model I started with disappeared. Mid-hackathon, gemini-2.5-flash started returning 404 for our new API key ("no longer available to new users"). I switched to gemini-3.5-flash and added an ordered fallback list, so a retired model, a timeout or a 5xx moves on to the next model automatically.

2. Free-tier rate limits. Scanning two photos and then running the safety check in quick succession hit the per-minute limit. Since each model has its own free-tier quota, a 429 now falls through to the backup model instead of showing an error:

if (res.status === 429) {
  lastError = new GeminiError("Gemini's free-tier rate limit was hit. Wait a minute and try again.", 429);
  continue; // try the next model; each has its own quota
}

3. "Loading…" forever on Vercel. Locally everything worked; deployed, the app hung. The cause was MongoDB Atlas's IP Access List, which only allowed my laptop. Two lessons: allow 0.0.0.0/0 for serverless hosts, and don't cache a failed connection promise. Ours was memoized, so even after fixing Atlas the warm function stayed broken:

clientPromise = new MongoClient(uri, { serverSelectionTimeoutMS: 8000 })
  .connect()
  .catch((e) => { clientPromise = null; throw e; }); // retry next request

4. "Take a photo" opened the file picker. On desktop, <input capture> is ignored. I replaced it with a real in-page camera using getUserMedia, a live <video> preview and a canvas capture, falling back to the file picker only when no camera is available.

5. Notifications silently failed on Android. new Notification() throws on Android Chrome. Routing reminders through a service worker's showNotification() fixed it, and added action buttons, so "I took it" works straight from the notification.

Designing for people who can't see the screen

Accessibility was a constraint from the start, not a feature added later:

Every action (scan, save, take, skip, snooze) has spoken feedback, not just a visual toast.
The tab list is keyboard-navigable, results are announced through aria-live regions, and the reminder is a proper alertdialog.
Text size, contrast and theme are user settings applied with data-* attributes on <html>, so the whole design system scales together.

Building for low vision made the app better for everyone.

Safety boundaries

DoseVoice helps people read and remember. It is deliberately not a doctor:

It never invents directions. If they're missing, it asks for another photo.
The interaction check only reports documented interactions, and frames each one as a question for your pharmacist, never "stop taking X".
Every screen says it's not medical advice.

What's next

Pill identification by shape and color, to catch mix-ups
Caregiver alerts when a dose is missed
Refill reminders from the "refills remaining" field
Multilingual speech
A full PWA with push reminders even when the browser is closed

What I learned

Structured output is what makes an LLM usable in high-stakes apps. The schema matters more than the prompt.
Plan for the model to change under you. Fallbacks and good errors paid off within hours.
Browser APIs are underrated. Camera, speech, and service workers gave us an app-like experience with no app store.
Learned how to deploy an app for free on Vercel and used it to make the app accessible to everyone.

Thanks to the HackUMass XIV organizers and MLH. If you or someone you know struggles with medication labels, give DoseVoice a try. Feedback welcome!

DEV Community: haricharanparna