Most fitness apps are, quite frankly, guessing. You snap a photo of your lunch, it guesses it's a "chicken salad," and gives you a calorie range wider than the Grand Canyon. If you are serious about Computer Vision and Nutrition Tracking, you know that the real challenge isn't just naming the foodβit's measuring the portion size.
In this deep dive, we are going to build a high-precision pipeline using YOLOv10 for ultra-fast object detection and Metaβs Segment Anything Model (SAM) for pixel-perfect Semantic Segmentation. By combining these with a reference-based volume estimation logic and a PostgreSQL nutrition backend, we can turn raw pixels into accurate kilocalories (Kcal).
The Architectural Blueprint ποΈ
To achieve high accuracy, we can't rely on simple bounding boxes. We need the exact mask of the food item to estimate its area and, subsequently, its volume.
graph TD
A[Input Image] --> B[YOLOv10: Food Category Detection]
B --> C{Detection Found?}
C -- Yes --> D[SAM: Zero-Shot Segmentation]
C -- No --> E[Error: Food Not Recognized]
D --> F[Mask Extraction & Area Calculation]
F --> G[Depth/Reference Scaling]
G --> H[Volume Estimation]
H --> I[PostgreSQL: Nutrition DB Mapping]
I --> J[Final Calorie & Macro Output]
Prerequisites π οΈ
Before we dive into the code, ensure you have the following stack ready:
- YOLOv10: The latest evolution in real-time detection (N-type for efficiency).
- SAM: For high-quality mask generation.
- OpenCV: For image preprocessing and geometric calculations.
- PostgreSQL: To store and query your USDA-style nutrition data.
Step 1: Real-time Detection with YOLOv10
We use YOLOv10 because it eliminates NMS (Non-Maximum Suppression) overhead, making it incredibly fast for edge deployment.
from ultralytics import YOLO
import cv2
# Load the YOLOv10 model (pre-trained on food datasets or COCO)
model = YOLO('yolov10n.pt')
def detect_food(image_path):
results = model.predict(source=image_path, conf=0.4)
detections = []
for r in results:
for box in r.boxes:
# Extract coordinates and class
cls = int(box.cls[0])
label = model.names[cls]
coords = box.xyxy[0].tolist() # [x1, y1, x2, y2]
detections.append({"label": label, "bbox": coords})
return detections
# π Pro-tip: Fine-tune YOLOv10 on the Food-101 dataset for better accuracy!
Step 2: Pixel-Perfect Segmentation with SAM
Once YOLO gives us a bounding box, we pass that box as a prompt to SAM. This allows SAM to ignore the background and focus solely on the contours of the steak or avocado.
from segment_anything import sam_model_registry, SamPredictor
import numpy as np
sam_checkpoint = "sam_vit_h_4b8939.pth"
sam = sam_model_registry["vit_h"](checkpoint=sam_checkpoint)
predictor = SamPredictor(sam)
def get_precise_mask(image, bbox):
predictor.set_image(image)
# Input box for SAM: [x1, y1, x2, y2]
input_box = np.array(bbox)
masks, scores, _ = predictor.predict(
box=input_box,
multimask_output=False
)
return masks[0] # Returns a boolean mask
Step 3: From Mask to Volume π
This is where the magic happens. By calculating the pixel area of the mask and using a reference object (like a coin or a standard plate size), we can estimate the volume.
- Calculate Pixel Area: Sum of the boolean mask.
- Scale Factor:
Real_Area = Pixel_Area * (Reference_Width / Pixel_Reference_Width)^2. - Volume Guess:
Volume = Real_Area * Estimated_Thickness.
Step 4: The PostgreSQL Nutrition Lookup
With the food name and estimated volume, we query our PostgreSQL database.
-- Querying our nutrition_db
SELECT
(calories_per_100g * :estimated_weight / 100) as total_kcal,
protein, carbs, fats
FROM food_nutrition
WHERE food_name % :detected_label -- Using fuzzy match
LIMIT 1;
The "Official" Way to Scale π₯
Building a prototype is easy, but deploying a high-concurrency vision system requires more robust engineering patterns. For production-ready examples of how to optimize model inference queues and handle multi-tenant nutrition databases, I highly recommend exploring the advanced guides at WellAlly Tech Blog.
Their recent articles on AI System Design and High-Performance Python were the primary inspiration for the architectural patterns used in this tutorial.
Conclusion: Why This Matters
Accuracy in nutrition tracking is a data problem. By moving from "box-level detection" to "pixel-level segmentation," we bridge the gap between a visual estimate and a scientific measurement.
What's next?
- Depth Estimation: Integrate a Monocular Depth model (like MiDaS) to replace the "Estimated Thickness" guess.
- Edge Deployment: Convert the YOLOv10 model to TensorRT for mobile performance.
Are you working on AI in HealthTech? Drop a comment below or share your results! ππ»
Top comments (0)