wellallyTech

Posted on Jul 17

From Pixels to Proteins: Building an AI Nutritionist with Segment Anything (SAM) and GPT-4o Vision 🥑

#ai #python #tutorial #opensource

Stop me if you’ve heard this before: "I’ll start tracking my macros today," only to give up three hours later because manually weighing a chicken breast is a logistical nightmare.

In the world of computer vision and multimodal AI, we are moving past simple classification. Today, we are building a system that doesn't just see a "pizza"—it calculates its volume, estimates its density, and predicts its macronutrients using the Segment Anything Model (SAM) and GPT-4o Vision. By combining precise object segmentation with the reasoning power of Large Multimodal Models (LMMs), we can turn a single photo into a comprehensive nutritional report.

The Architecture: How It Works 🏗️

To achieve high accuracy, we can't just throw a messy photo at an LLM. We need a pipeline that isolates the food, calculates its physical dimensions, and then asks GPT-4o to perform "informed reasoning."

graph TD
    A[Raw RGB-D Image] --> B[OpenCV Pre-processing]
    B --> C{Segment Anything Model}
    C -->|Generate Masks| D[Food Instance Isolation]
    D --> E[Depth-based Volume Estimation]
    E --> F[GPT-4o Vision API]
    G[Nutritional Database/Context] --> F
    F --> H[Final Nutrition Label: Calories, Protein, Carbs, Fats]

Prerequisites 🛠️

Before we dive into the code, ensure you have the following stack ready:

PyTorch: For running the SAM encoder.
Segment Anything (SAM): Specifically the vit_h or vit_l weights.
OpenCV: For image manipulation.
OpenAI SDK: To access GPT-4o Vision.
Hardware: A GPU with at least 8GB VRAM is recommended for local inference of SAM.

Step 1: Precise Segmentation with SAM

Traditional bounding boxes include too much noise (like the plate or the table). SAM allows us to extract the exact mask of the food item.

import torch
import cv2
import numpy as np
from segment_anything import sam_model_registry, SamPredictor

# Initialize SAM
sam_checkpoint = "sam_vit_h_4b8939.pth"
model_type = "vit_h"
device = "cuda" if torch.cuda.is_available() else "cpu"

sam = sam_model_registry[model_type](checkpoint=sam_checkpoint)
sam.to(device=device)
predictor = SamPredictor(sam)

def get_food_mask(image, input_point):
    predictor.set_image(image)
    masks, scores, logits = predictor.predict(
        point_coords=np.array([input_point]),
        point_labels=np.array([1]),
        multimask_output=True,
    )
    # We take the mask with the highest score
    return masks[np.argmax(scores)]

Step 2: From Masks to Volume 📏

If you are using a depth camera (like an Intel RealSense or iPhone LiDAR), you can map the pixels in your mask to 3D space. By calculating the area of the mask and multiplying it by the average height relative to the plate, we get a volumetric estimate ($V = \sum \text{depth}_{pixel}$).

If you're using a standard 2D image, we provide the "reference object" (like a coin or the plate size) to GPT-4o to help it infer scale.

Step 3: Multimodal Reasoning with GPT-4o

Now we send the isolated food image and our calculated volume data to GPT-4o. This is where the magic happens. We use Structured Outputs to ensure the AI returns valid JSON.

import openai

def analyze_nutrition(image_path, volume_cm3):
    client = openai.OpenAI()

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": f"Analyze this food. Estimated volume: {volume_cm3}cm3. Provide calories and macros (P/C/F) in JSON format."},
                    {
                        "type": "image_url",
                        "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}
                    },
                ],
            }
        ],
        response_format={ "type": "json_object" }
    )
    return response.choices[0].message.content

Implementing Production-Ready Patterns 🚀

While this script works for a hobby project, moving this to production requires handling edge cases like overlapping food items, lighting variance, and API latency.

For a deeper dive into Production-grade AI Architectures and advanced computer vision optimizations, I highly recommend checking out the engineering deep-dives at WellAlly Tech Blog. They cover how to scale these multimodal pipelines and optimize inference costs which is crucial when you're dealing with high-resolution vision tokens.

The Results: Better Data, Better Health

By using SAM, we reduce the "background noise" that often confuses vision models. Instead of the AI seeing "a table with a plate of pasta," it sees "150g of Penne with 40g of Pesto."

Why this works:

Isolation: SAM removes the plate, which can vary in size and color.
Context: GPT-4o understands that a 200cm³ block of cheese is significantly more caloric than 200cm³ of lettuce.
Speed: Recent optimizations in SAM-HQ make this pipeline run in near real-time.

Conclusion 🌯

We are standing at the intersection of Physical Sensing and Semantic Understanding. Building an AI that understands the caloric density of your lunch is just the beginning.

Are you working on Vision-Language Models (VLMs)? Have you tried fine-tuning SAM for specific food textures? Let’s chat in the comments! 🚀

If you enjoyed this tutorial, don't forget to ❤️ and bookmark! For more advanced AI patterns, visit wellally.tech/blog.

DEV Community