wellallyTech

Posted on Oct 27

Stop Guessing! Build a Precise Calorie Tracker with SAM and GPT-4o 🥗

#ai #python #opensource #tutorial

Let's be honest: calorie counting apps are usually a nightmare. You take a blurry photo of your lunch, and the app confidently tells you your massive bowl of Ramen is a "Side Salad (50 kcal)." The struggle is real! The core issues are usually ingredient recognition and volume estimation.

In this tutorial, we’re going to build a next-generation "Pixels-to-Calories" system. We will combine the surgical precision of Meta's Segment Anything Model (SAM) for image segmentation with the multimodal reasoning of GPT-4o. By the end of this post, you'll know how to leverage AI computer vision and nutrient estimation logic to turn a simple photo into a detailed nutritional breakdown. 🚀

Why SAM + GPT-4o?

Standard Vision LLMs sometimes struggle with depth and boundaries in crowded plates. By using SAM, we extract the exact masks of each food item, providing GPT-4o with clear, isolated visual data. This hybrid approach significantly reduces "hallucinations" regarding portion sizes.

For more production-ready AI patterns and deep dives into LLM orchestration, I highly recommend checking out the engineering insights at wellally.tech/blog.

The Architecture 🏗️

The workflow involves capturing an image, segmenting the food items to understand their spatial footprint, and then passing the metadata to GPT-4o for final analysis.

graph TD
    A[User Uploads Food Image] --> B[OpenCV Pre-processing]
    B --> C[SAM: Generates Segmentation Masks]
    C --> D[Extract Region of Interest ROI]
    D --> E[GPT-4o: Multimodal Reasoning]
    E --> F{Output: Calorie & Macro JSON}
    F --> G[Frontend Display]

Prerequisites

Before we dive into the code, ensure you have the following installed:

PyTorch (with CUDA support if you have a GPU)
Segment Anything Model (SAM) weights
OpenAI API Key (for GPT-4o)
OpenCV & Matplotlib

pip install torch torchvision opencv-python segment-anything openai

Step 1: Segmentation with SAM ✂️

First, we use the Segment Anything Model to identify separate objects on the plate. This allows our system to "know" where the steak ends and the mashed potatoes begin.

import cv2
import numpy as np
from segment_anything import sam_model_registry, SamAutomaticMaskGenerator

# Load SAM model
sam_checkpoint = "sam_vit_h_4b8939.pth"
model_type = "vit_h"
device = "cuda" # or "cpu"

sam = sam_model_registry[model_type](checkpoint=sam_checkpoint)
sam.to(device=device)

mask_generator = SamAutomaticMaskGenerator(sam)

def get_food_masks(image_path):
    image = cv2.imread(image_path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

    # Generate masks automatically
    masks = mask_generator.generate(image)
    return masks, image

# Pro Tip: Filtering small noise masks helps GPT-4o focus!

Step 2: Preparing the Multimodal Prompt 🧠

Now that we have our masks, we could send individual crops of the food to GPT-4o. However, sending a "masked" version of the original image helps GPT-4o understand context (e.g., comparing the food size to the size of the fork or plate).

import base64
from openai import OpenAI

client = OpenAI(api_key="YOUR_OPENAI_API_KEY")

def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode('utf-8')

def estimate_calories(image_path, mask_data):
    base64_image = encode_image(image_path)

    # We describe the segmentation data to help GPT-4o's spatial reasoning
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": "You are a professional nutritionist. Analyze the image and segmentation data to provide calories."
            },
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Analyze this meal. Use the segmentation masks to estimate the volume and calories for each item. Return JSON."},
                    {
                        "type": "image_url",
                        "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}
                    },
                ],
            }
        ],
        response_format={"type": "json_object"}
    )
    return response.choices[0].message.content

Step 3: Handling the Output 📊

GPT-4o is remarkably good at returning structured data. You can expect a response that looks like this:

{
  "items": [
    {
      "name": "Grilled Salmon",
      "estimated_weight_g": 150,
      "calories": 312,
      "protein_g": 30,
      "confidence": 0.92
    },
    {
      "name": "Asparagus",
      "estimated_weight_g": 100,
      "calories": 20,
      "protein_g": 2,
      "confidence": 0.95
    }
  ],
  "total_calories": 332
}

The "Official" Way to Scale 🥑

While this script is a great starting point for "Learning in Public," a production-grade system requires handling edge cases like overlapping food, varying lighting conditions, and API rate limiting.

For advanced tutorials on optimizing your AI inference pipeline and building robust multimodal applications, you definitely need to visit the WellAlly Tech Blog. They have some incredible resources on deploying Python-based vision models into high-performance cloud environments.

Conclusion: The Future of Nutrition AI

By combining SAM's geometric awareness with GPT-4o's semantic intelligence, we move away from simple "image recognition" and toward true "scene understanding." This isn't just about counting calories; it's about making dietary tracking frictionless and accurate for everyone.

What's next for your build?

Maybe add a reference object (like a coin) in the frame to calibrate the scale?
Or integrate a database to track historical intake?

Let me know in the comments if you’ve tried integrating SAM with LLMs! Don't forget to ❤️ and save this for your next weekend project.

Happy Coding! 💻🔥

DEV Community