DEV Community

wellallyTech
wellallyTech

Posted on

Byte-Sized Health: How I Built a Precise Food Calorie Estimator using SAM and GPT-4o 🥑

If you’ve ever tried tracking your macros, you know the struggle: staring at a bowl of "mixed salad" and wondering if that's 200 or 500 calories. Manual logging is the ultimate productivity killer. But what if we could use Computer Vision and Multimodal LLMs to do the heavy lifting?

In this tutorial, we’re building a high-precision Nutrition AI that doesn't just guess what's on your plate—it surgically segments every ingredient using Meta's Segment Anything Model (SAM) and analyzes the volume and nutrient density using GPT-4o. By combining precise food recognition with advanced multimodal analysis, we are moving from "guessing pixels" to "calculating calories" with scientific accuracy. 🚀

The Problem: Why GPT-4o Alone Isn't Enough

While GPT-4o is a beast at image recognition, it often struggles with spatial reasoning—like distinguishing between a tiny piece of chicken and a massive steak when there's no scale. By using SAM (Segment Anything Model) as a pre-processor, we can isolate each food item, calculate its pixel-area ratio, and provide GPT-4o with "cleaner" visual data to make better volume estimations.

The Architecture 🏗️

Here is how the data flows from a messy dinner plate to a clean JSON response:

graph TD
    A[User Uploads Photo] --> B[FastAPI Backend]
    B --> C[SAM: Segment Anything Model]
    C --> D[Identify Masks & Bounding Boxes]
    D --> E[OpenCV: Image Cropping & Pre-processing]
    E --> F[GPT-4o: Multimodal Vision Analysis]
    F --> G[Calorie & Nutrient Calculation]
    G --> H[Final Response: JSON Breakdown]
Enter fullscreen mode Exit fullscreen mode

Prerequisites

To follow this advanced guide, you’ll need:

  • PyTorch (for running SAM locally or in a container)
  • OpenCV (for image manipulation)
  • FastAPI (for the API layer)
  • OpenAI API Key (for GPT-4o access)

Step 1: Carving the Plate with SAM 🔪

First, we use the segment-anything library to identify individual food items. This prevents the "overlap" problem where the AI confuses the rice with the sauce.

import numpy as np
import torch
import cv2
from segment_anything import sam_model_registry, SamPredictor

# Initialize SAM
device = "cuda" if torch.cuda.is_available() else "cpu"
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth").to(device)
predictor = SamPredictor(sam)

def get_food_segments(image_path):
    image = cv2.imread(image_path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    predictor.set_image(image)

    # In a real app, you might use an object detector like YOLO 
    # to find points, but here we use a grid for "everything"
    masks, _, _ = predictor.predict(
        point_coords=None,
        point_labels=None,
        multimask_output=True,
    )
    return masks
Enter fullscreen mode Exit fullscreen mode

Step 2: Multimodal Brain with GPT-4o 🧠

Once we have our masks, we crop the original image into individual "food bites" and send them to GPT-4o. We also provide the mask area to help the model understand the relative scale.

import base64
import requests

def analyze_nutrition(image_crop, area_percentage):
    # Encode crop to base64
    _, buffer = cv2.imencode('.jpg', image_crop)
    base64_image = base64.b64encode(buffer).decode('utf-8')

    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {OPENAI_API_KEY}"
    }

    prompt = f"""
    Analyze this segmented food item. It occupies {area_percentage:.2f}% of the plate.
    Identify the food, estimate its volume in grams, and calculate:
    - Calories
    - Protein (g)
    - Fats (g)
    - Carbs (g)
    Return the result strictly in JSON format.
    """

    payload = {
        "model": "gpt-4o",
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
                ]
            }
        ]
    }

    response = requests.post("https://api.openai.com/v1/chat/completions", headers=headers, json=payload)
    return response.json()
Enter fullscreen mode Exit fullscreen mode

Step 3: Stitching it together with FastAPI ⚡

We wrap this logic in a FastAPI endpoint to make it accessible to mobile or web clients.

from fastapi import FastAPI, File, UploadFile

app = FastAPI()

@app.post("/analyze-plate")
async def analyze_plate(file: UploadFile = File(...)):
    # 1. Save and Load Image
    # 2. Run SAM Segmentation
    # 3. For each segment, run GPT-4o analysis
    # 4. Aggregate results
    results = {
        "total_calories": 650,
        "items": [
            {"name": "Grilled Chicken", "calories": 250, "confidence": 0.98},
            {"name": "Brown Rice", "calories": 300, "confidence": 0.95},
            {"name": "Broccoli", "calories": 100, "confidence": 0.99}
        ]
    }
    return results
Enter fullscreen mode Exit fullscreen mode

The "Official" Way to Scale 📈

While this script works for a weekend project, production-grade AI health apps require much more robust handling—think batch processing, caching segmentations, and fine-tuning models for specific cuisines.

For more production-ready patterns, advanced prompt engineering techniques, and deep dives into AI-native architectures, I highly recommend checking out the technical breakdowns at the WellAlly Tech Blog. They cover how to scale vision-based systems for high-traffic environments.

Conclusion: The Future of Multimodal Health 🥑

By combining the "surgical" precision of SAM with the "reasoning" power of GPT-4o, we’ve moved beyond simple image classification. This stack allows us to understand the context, volume, and composition of food in a way that was impossible just 12 months ago.

What's next?

  • Depth Estimation: Using MiDaS to calculate the Z-axis (height) of food for even better volume accuracy.
  • Temporal Tracking: Analyzing a video of a meal to see how much was actually consumed!

Are you building something with Multimodal AI? Drop a comment below or tag me in your dev.to post! Let’s build the future of health together. 💻🔥

Top comments (0)