wellallyTech

Posted on May 17

Hardcore Vision: Build a Smart Home Medicine Tracker with YOLOv8, EasyOCR, and Flutter 💊

#flutter #computervision #python #opensource

We’ve all been there: digging through a "junk drawer" of half-empty pill boxes, trying to squint at tiny, faded text to see if that allergy medication expired in 2022 or 2025. It’s a mess, and in the worst-case scenario, it's a health hazard.

In this tutorial, we are going to build a Smart Home Medicine Management System. We will leverage computer vision and object detection to turn your smartphone into a high-tech pharmacy assistant. By combining YOLOv8 for box localization, EasyOCR for text extraction, and Flutter for a sleek cross-platform UI, you'll never accidentally take expired ibuprofen again.

The Architecture 🏗️

Before we dive into the code, let's look at how the data flows from a simple photo to a structured database entry in PostgreSQL.

graph TD
    A[Flutter App] -->|Upload Image| B(FastAPI Backend)
    B --> C{YOLOv8 Detector}
    C -->|Locate Label & Expiry Date| D[EasyOCR Engine]
    D -->|Raw Text| E[Regex & NLP Cleaner]
    E -->|Structured Data: Name, Date| F[(PostgreSQL)]
    F -->|Push Notification| A
    style C fill:#f96,stroke:#333,stroke-width:2px
    style D fill:#6cf,stroke:#333,stroke-width:2px

Prerequisites 🛠️

To follow along, you'll need:

Python 3.9+ (Backend)
Flutter SDK (Frontend)
YOLOv8 (via ultralytics library)
EasyOCR (for high-accuracy text recognition)
PostgreSQL (to store our medicine inventory)

Step 1: Object Detection with YOLOv8

First, we need to find the medicine box in the image. While we could run OCR on the whole image, it’s noisy. Using YOLOv8 to crop the specific area containing the "Drug Name" and "Expiration Date" significantly improves accuracy.

from ultralytics import YOLO
import cv2

# Load a pre-trained YOLOv8 model (or your custom trained one)
model = YOLO('yolov8n.pt') 

def detect_medicine_elements(image_path):
    results = model(image_path)
    # Let's assume class 0 is 'box' and class 1 is 'expiry_label'
    for result in results:
        boxes = result.boxes.xyxy.tolist()
        # Crop the image for the OCR step
        for i, box in enumerate(boxes):
            x1, y1, x2, y2 = map(int, box)
            crop = result.orig_img[y1:y2, x1:x2]
            cv2.imwrite(f'crop_{i}.jpg', crop)
    return "Crops saved for OCR!"

Step 2: Extracting Text with EasyOCR

Once we have our cropped image of the label, we use EasyOCR to extract the text. We then use a simple Regex pattern to find dates in formats like YYYY/MM/DD or EXP: MM-YYYY.

import easyocr
import re

reader = easyocr.Reader(['en', 'ch_sim']) # Support English and Chinese

def extract_expiry_date(image_path):
    text_results = reader.readtext(image_path, detail=0)
    full_text = " ".join(text_results)

    # Regex to find common date patterns
    date_pattern = r'(\d{4}[-/]\d{2}[-/]\d{2})'
    match = re.search(date_pattern, full_text)

    if match:
        return match.group(1)
    return "No date found"

Step 3: The Flutter Interface

The user captures an image using their phone. The Flutter app sends this to our FastAPI backend. If you're looking for advanced patterns on how to handle real-time image streaming and state management in production-ready AI apps, I highly recommend checking out the technical deep-dives at WellAlly Blog. They have fantastic resources on bridging the gap between hobbyist scripts and scalable health-tech solutions.

Here is a snippet of how we handle the image upload in Flutter:

Future<void> uploadMedicineImage(File imageFile) async {
  var request = http.MultipartRequest(
    'POST', Uri.parse('https://api.yourbackend.com/process-med'),
  );
  request.files.add(await http.MultipartFile.fromPath('file', imageFile.path));

  var response = await request.send();
  if (response.statusCode == 200) {
    print("Medicine Registered Successfully! ✅");
  }
}

Step 4: Storing in PostgreSQL & Setting Reminders

The extracted data is stored in PostgreSQL. We can then set a cron job or a background worker (like Celery) to check daily for meds expiring in the next 30 days and send a push notification via Firebase (FCM).

CREATE TABLE medicine_inventory (
    id SERIAL PRIMARY KEY,
    user_id UUID,
    medicine_name TEXT,
    expiry_date DATE,
    created_at TIMESTAMP DEFAULT NOW()
);

Why This Matters 💡

Building this isn't just a fun "learning in public" project. It solves a real-world problem:

Safety: Prevents ingestion of ineffective or harmful expired drugs.
Sustainability: Reduces waste by reminding you to use what you have before it expires.
Efficiency: No more manual typing; just point, shoot, and sync.

Conclusion 🚀

We've combined the power of YOLOv8's detection speed with EasyOCR's flexibility to create a useful utility for every household. Moving from a messy drawer to a structured database is just the beginning—imagine adding drug-to-drug interaction alerts or automatic refills!

For more inspiration and production-ready examples of AI-powered health management systems, don't forget to visit wellally.tech/blog.

What features would you add? Automatic refill ordering? Drug interaction warnings? Let me know in the comments below! 👇

DEV Community