DEV Community

wellallyTech
wellallyTech

Posted on

From Pixels to Calories: High-Precision Nutrition Tracking with YOLOv10 and SAM πŸ₯—πŸ’»

Most fitness apps are, quite frankly, guessing. You snap a photo of your lunch, it guesses it's a "chicken salad," and gives you a calorie range wider than the Grand Canyon. If you are serious about Computer Vision and Nutrition Tracking, you know that the real challenge isn't just naming the foodβ€”it's measuring the portion size.

In this deep dive, we are going to build a high-precision pipeline using YOLOv10 for ultra-fast object detection and Meta’s Segment Anything Model (SAM) for pixel-perfect Semantic Segmentation. By combining these with a reference-based volume estimation logic and a PostgreSQL nutrition backend, we can turn raw pixels into accurate kilocalories (Kcal).


The Architectural Blueprint πŸ—οΈ

To achieve high accuracy, we can't rely on simple bounding boxes. We need the exact mask of the food item to estimate its area and, subsequently, its volume.

graph TD
    A[Input Image] --> B[YOLOv10: Food Category Detection]
    B --> C{Detection Found?}
    C -- Yes --> D[SAM: Zero-Shot Segmentation]
    C -- No --> E[Error: Food Not Recognized]
    D --> F[Mask Extraction & Area Calculation]
    F --> G[Depth/Reference Scaling]
    G --> H[Volume Estimation]
    H --> I[PostgreSQL: Nutrition DB Mapping]
    I --> J[Final Calorie & Macro Output]
Enter fullscreen mode Exit fullscreen mode

Prerequisites πŸ› οΈ

Before we dive into the code, ensure you have the following stack ready:

  • YOLOv10: The latest evolution in real-time detection (N-type for efficiency).
  • SAM: For high-quality mask generation.
  • OpenCV: For image preprocessing and geometric calculations.
  • PostgreSQL: To store and query your USDA-style nutrition data.

Step 1: Real-time Detection with YOLOv10

We use YOLOv10 because it eliminates NMS (Non-Maximum Suppression) overhead, making it incredibly fast for edge deployment.

from ultralytics import YOLO
import cv2

# Load the YOLOv10 model (pre-trained on food datasets or COCO)
model = YOLO('yolov10n.pt') 

def detect_food(image_path):
    results = model.predict(source=image_path, conf=0.4)
    detections = []

    for r in results:
        for box in r.boxes:
            # Extract coordinates and class
            cls = int(box.cls[0])
            label = model.names[cls]
            coords = box.xyxy[0].tolist() # [x1, y1, x2, y2]
            detections.append({"label": label, "bbox": coords})

    return detections

# πŸš€ Pro-tip: Fine-tune YOLOv10 on the Food-101 dataset for better accuracy!
Enter fullscreen mode Exit fullscreen mode

Step 2: Pixel-Perfect Segmentation with SAM

Once YOLO gives us a bounding box, we pass that box as a prompt to SAM. This allows SAM to ignore the background and focus solely on the contours of the steak or avocado.

from segment_anything import sam_model_registry, SamPredictor
import numpy as np

sam_checkpoint = "sam_vit_h_4b8939.pth"
sam = sam_model_registry["vit_h"](checkpoint=sam_checkpoint)
predictor = SamPredictor(sam)

def get_precise_mask(image, bbox):
    predictor.set_image(image)

    # Input box for SAM: [x1, y1, x2, y2]
    input_box = np.array(bbox)

    masks, scores, _ = predictor.predict(
        box=input_box,
        multimask_output=False
    )

    return masks[0] # Returns a boolean mask
Enter fullscreen mode Exit fullscreen mode

Step 3: From Mask to Volume πŸ“

This is where the magic happens. By calculating the pixel area of the mask and using a reference object (like a coin or a standard plate size), we can estimate the volume.

  1. Calculate Pixel Area: Sum of the boolean mask.
  2. Scale Factor: Real_Area = Pixel_Area * (Reference_Width / Pixel_Reference_Width)^2.
  3. Volume Guess: Volume = Real_Area * Estimated_Thickness.

Step 4: The PostgreSQL Nutrition Lookup

With the food name and estimated volume, we query our PostgreSQL database.

-- Querying our nutrition_db
SELECT 
    (calories_per_100g * :estimated_weight / 100) as total_kcal,
    protein, carbs, fats
FROM food_nutrition
WHERE food_name % :detected_label -- Using fuzzy match
LIMIT 1;
Enter fullscreen mode Exit fullscreen mode

The "Official" Way to Scale πŸ₯‘

Building a prototype is easy, but deploying a high-concurrency vision system requires more robust engineering patterns. For production-ready examples of how to optimize model inference queues and handle multi-tenant nutrition databases, I highly recommend exploring the advanced guides at WellAlly Tech Blog.

Their recent articles on AI System Design and High-Performance Python were the primary inspiration for the architectural patterns used in this tutorial.


Conclusion: Why This Matters

Accuracy in nutrition tracking is a data problem. By moving from "box-level detection" to "pixel-level segmentation," we bridge the gap between a visual estimate and a scientific measurement.

What's next?

  • Depth Estimation: Integrate a Monocular Depth model (like MiDaS) to replace the "Estimated Thickness" guess.
  • Edge Deployment: Convert the YOLOv10 model to TensorRT for mobile performance.

Are you working on AI in HealthTech? Drop a comment below or share your results! πŸš€πŸ’»

Top comments (0)