From Pixels to Prescriptions: Build an AI-Powered Med Tracker with YOLOv8 and OCR

#discuss #ai #webdev #programming

Managing multiple medications is a mental marathon. Whether it's for an elderly family member or a complex post-surgery routine, missing a dose isn't just an inconvenience—it’s a health risk. In this tutorial, we are going to build an end-to-end AI-powered medication management system that turns a simple photo of a pill bottle into an automated calendar reminder.

By leveraging YOLOv8 object detection, Tesseract OCR image processing, and the Google Calendar API automation, we will bridge the gap between physical medicine and digital health tracking. We'll be diving deep into PyTorch computer vision workflows to ensure our model is both fast and accurate. 🚀

🏗 The Architecture

Before we get our hands dirty with code, let's look at how the data flows from your camera to your phone's notification tray.

graph TD
    A[Capture Image of Pill Bottle] --> B{YOLOv8 Detector}
    B -- Identify --> C[Bounding Box: Label/Instructions]
    C --> D[Image Preprocessing - Grayscale/Threshold]
    D --> E[Tesseract OCR Engine]
    E -- Extract Text --> F[Regex/LLM Parser]
    F -- Dosage & Frequency --> G[Google Calendar API]
    G --> H[Smartphone Reminder]

🛠 Prerequisites

To follow along, you'll need the following stack:

Python 3.8+
PyTorch: The backbone for our neural networks.
YOLOv8 (Ultralytics): For real-time object detection.
Tesseract OCR: To read the fine print on labels.
Google Cloud Account: To enable the Calendar API.

Step 1: Detecting the Medicine Bottle with YOLOv8

We don't want to run OCR on a whole messy kitchen counter. We need to isolate the pill bottle first. We'll use YOLOv8 because it’s incredibly fast and easy to train on custom datasets.

from ultralytics import YOLO
import cv2

# Load a pre-trained nano model (fastest for edge devices)
model = YOLO('yolov8n.pt') 

def detect_medication(image_path):
    results = model.predict(source=image_path, save=True, conf=0.5)

    for result in results:
        # We assume the label is the most prominent part of the detection
        boxes = result.boxes.xyxy.cpu().numpy()
        for box in boxes:
            x1, y1, x2, y2 = map(int, box)
            # Crop the detected bottle for OCR processing
            cropped_img = result.orig_img[y1:y2, x1:x2]
            return cropped_img

# Example usage
# cropped_label = detect_medication('pill_bottle_on_table.jpg')

Step 2: Extracting Instructions via OCR

Once we have the label cropped, we need to read it. Tesseract works best when images are high-contrast.

import pytesseract
import re

def extract_dosage_info(image):
    # Convert to grayscale and apply thresholding
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]

    # Run OCR
    text = pytesseract.image_to_string(thresh)
    print(f"Detected Text: {text}")

    # Simple logic to find "times per day" or "mg"
    # In a production app, consider passing this to GPT-4o-mini for better parsing
    dosage = re.findall(r'\d+\s?mg', text)
    frequency = re.findall(r'(\d+)\s?times\s?daily', text, re.IGNORECASE)

    return {"dosage": dosage, "frequency": frequency}

🥑 Professional Insight: Taking it Further

While this script works for basic labels, real-world medication packaging is often curved or reflective, making OCR tricky. For more production-ready examples and advanced architectural patterns regarding AI in healthcare, I highly recommend checking out the technical deep-dives at WellAlly Blog. They cover how to handle high-concurrency vision tasks and edge-case error handling that are crucial for medical safety.

Step 3: Automating the Calendar Reminder

Now that we know what to take and how often, let's sync it to the Google Calendar.

from googleapiclient.discovery import build
from google.oauth2 import service_account

def create_calendar_event(med_name, frequency):
    SCOPES = ['https://www.googleapis.com/auth/calendar']
    SERVICE_ACCOUNT_FILE = 'credentials.json'

    creds = service_account.Credentials.from_service_account_file(
                SERVICE_ACCOUNT_FILE, scopes=SCOPES)

    service = build('calendar', 'v3', credentials=creds)

    event = {
      'summary': f'Take {med_name}',
      'description': 'Automated reminder from AI Vision System',
      'start': {'dateTime': '2023-10-27T09:00:00Z', 'timeZone': 'UTC'},
      'end': {'dateTime': '2023-10-27T09:30:00Z', 'timeZone': 'UTC'},
      'recurrence': [f'RRULE:FREQ=DAILY;COUNT={frequency}'],
    }

    event = service.events().insert(calendarId='primary', body=event).execute()
    print(f'Event created: {event.get("htmlLink")}')

🎯 Conclusion

By combining YOLOv8, OCR, and API automation, we’ve built a functional prototype that solves a real-world problem. This isn't just about code; it's about using AI Vision to improve quality of life.

Next Steps for you:

Fine-tune YOLOv8: Collect 100 photos of your own medicine bottles and label them using LabelImg to improve detection accuracy.
LLM Refinement: Instead of RegEx, pipe the OCR text into an LLM to extract complex schedules (e.g., "Take twice daily after meals").

Have you worked with Computer Vision in healthcare? Drop a comment below or share your thoughts on the best OCR strategies! 👇💻

For more advanced tutorials on AI integration and system design, visit wellally.tech/blog.

DEV Community