DEV Community

Beck_Moulton
Beck_Moulton

Posted on

From Pixel to Protein: Automating My Diet with GPT-4o-mini and Segment Anything (SAM)

#ai #tutorial #python #webdev

Let’s be honest: manual diet logging is where fitness goals go to die. Tracking every almond and weighing every chicken breast is a full-time job that nobody wants. But what if we could combine Computer Vision, the Segment Anything Model (SAM), and the reasoning power of GPT-4o-mini to turn a single photo into a detailed nutritional breakdown?

In this tutorial, we’ll build a high-precision Automated Nutrition Tracking pipeline. We will leverage GPT-4o-mini for multimodal reasoning and SAM for precise spatial segmentation, solving the "depth and volume" estimation problem that plagues standard 2D image analysis. By the end of this post, you'll have a functional Nutrition AI API capable of identifying food items and estimating macros with impressive accuracy.

The Architecture 🏗️

The biggest challenge in visual food analysis isn't just identifying the food; it's understanding the quantity. We use SAM to isolate individual food components and then pass these segments to GPT-4o-mini for volumetric estimation and nutrient calculation.

graph TD
    A[User Uploads Food Image] --> B[OpenCV Pre-processing]
    B --> C[SAM: Segment Anything]
    C --> D[Extract Masks & Bounding Boxes]
    D --> E[GPT-4o-mini: Multimodal Analysis]
    E --> F[Macro & Calorie Estimation]
    F --> G[FastAPI Response: Nutrient JSON]
    G --> H[Final Nutrition Log]
Enter fullscreen mode Exit fullscreen mode

Prerequisites 🛠️

To follow along, you'll need:

  • Python 3.10+
  • FastAPI (for the web framework)
  • OpenAI API Key (for GPT-4o-mini)
  • Segment Anything (SAM) Weights (Facebook Research)
  • OpenCV (for image manipulation)

Step 1: Precision Segmentation with SAM

Traditional object detection just gives us a box. SAM gives us the exact pixels. This is crucial for distinguishing between a small portion of rice and a large one.

import numpy as np
import cv2
from segment_anything import sam_model_registry, SamPredictor

# Initialize SAM
sam_checkpoint = "sam_vit_h_4b8939.pth"
model_type = "vit_h"
sam = sam_model_registry[model_type](checkpoint=sam_checkpoint)
predictor = SamPredictor(sam)

def get_food_segments(image_path):
    image = cv2.imread(image_path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    predictor.set_image(image)

    # In a production scenario, we'd use a prompt or a grid to find masks
    # For now, let's assume we are targeting the primary objects
    masks, _, _ = predictor.predict(
        point_coords=None,
        point_labels=None,
        multimask_output=True,
    )
    return masks
Enter fullscreen mode Exit fullscreen mode

Step 2: Multimodal Reasoning with GPT-4o-mini

Once we have the segments, we send the image and the spatial context to GPT-4o-mini. Its ability to process vision and text simultaneously allows it to "guess" the weight based on common plate sizes and object proportions.

import base64
from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class NutritionInfo(BaseModel):
    food_name: str
    estimated_weight_g: float
    calories: int
    protein_g: float
    carbs_g: float
    fats_g: float

def analyze_nutrition(image_base64):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Identify the food in this image. Estimate the weight in grams and provide nutritional info in JSON format."},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"}}
                ],
            }
        ],
        response_format={"type": "json_object"}
    )
    return response.choices[0].message.content
Enter fullscreen mode Exit fullscreen mode

Step 3: Wrapping it in FastAPI 🚀

We need an endpoint to glue everything together. FastAPI is perfect for this because of its speed and native support for Pydantic.

from fastapi import FastAPI, UploadFile, File

app = FastAPI()

@app.post("/analyze-plate")
async def analyze_plate(file: UploadFile = File(...)):
    # 1. Read image
    contents = await file.read()

    # 2. (Optional) Run SAM for spatial validation
    # masks = get_food_segments(contents)

    # 3. GPT-4o-mini Analysis
    base64_image = base64.b64encode(contents).decode('utf-8')
    nutrition_data = analyze_nutrition(base64_image)

    return {"status": "success", "data": nutrition_data}
Enter fullscreen mode Exit fullscreen mode

The "Official" Way: Leveling Up Your AI Strategy 🥑

While this DIY pipeline is great for a prototype, building a production-grade health-tech application requires more robust handling of edge cases, such as overlapping food items or low-light conditions.

If you are looking for more advanced architectural patterns, production-ready AI pipelines, or deep dives into multimodal LLM deployment, I highly recommend exploring the insights over at the WellAlly Blog. It's a fantastic resource for developers who want to move past the "Hello World" of AI and into scalable, real-world systems.

Conclusion

By combining the spatial precision of SAM with the multimodal intelligence of GPT-4o-mini, we’ve bridged the gap between raw pixels and actionable nutritional data. This pipeline isn't just about counting calories; it's about reducing the friction between humans and their health goals.

What's next?

  • Add a reference object (like a coin) in the photo to provide SAM with a physical scale.
  • Integrate with the HealthKit API to sync logs directly to your phone.

Happy coding! If you enjoyed this build, don't forget to ❤️ and follow for more "Learning in Public" AI content! 🚀💻

Top comments (0)