wellallyTech

Posted on Sep 10

From Pixels to Proteins: Building a Precise AI Nutrition Tracker with GPT-4o and SAM 🥑

#ai #webdev #python #machinelearning

Have you ever tried logging your meals manually? It’s a nightmare. You're staring at a bowl of poke, trying to guess if that's 50g or 80g of salmon while fumbling through a database of 10,000 entries.

In this tutorial, we’re going to solve this using a cutting-edge Multimodal AI pipeline. We will combine the precision of the Segment Anything Model (SAM) for object localization with the reasoning power of GPT-4o vision and the ground-truth data from the USDA FoodData API. This is the ultimate guide to building an automated nutrition tracking system that actually works. 🚀

The Architecture 🏗️

The logic is simple but powerful: we use SAM to "cut out" the food items so the AI isn't confused by the background, then let GPT-4o identify the portions, and finally validate the nutrients against official databases.

graph TD
    A[User Uploads Food Image] --> B[SAM: Segment Individual Items]
    B --> C[GPT-4o: Identify Food & Estimate Mass/Volume]
    C --> D[USDA FoodData API: Fetch Exact Macros]
    D --> E[FastAPI: Aggregated Nutrients JSON]
    E --> F[User Dashboard]

Prerequisites 🛠️

To follow along, you’ll need:

Python 3.9+
GPT-4o API Key (via OpenAI)
Segment Anything (SAM) weights (sam_vit_h)
USDA FoodData API Key (Free to register)
FastAPI for the backend

Step 1: Precision Segmentation with SAM

Traditional vision models often struggle with overlapping food items. By using Meta's Segment Anything Model (SAM), we can generate masks for every distinct object in the image. This "pre-processing" helps GPT-4o focus on one ingredient at a time.

import numpy as np
from segment_anything import SamPredictor, sam_model_registry

# Load SAM model
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)

def get_food_segments(image):
    predictor.set_image(image)
    # Automatically generate masks for the entire image
    # For simplicity, we use the middle of the image as a prompt 
    # or use an automatic mask generator
    masks, scores, logits = predictor.predict(
        point_coords=np.array([[500, 500]]), 
        point_labels=np.array([1]),
        multimask_output=True
    )
    return masks[np.argmax(scores)]

Step 2: Identification & Estimation with GPT-4o

Now that we have segmented regions, we pass the original image + the mask information to GPT-4o. We don't just ask "What is this?"; we ask for a structured JSON response including estimated weight in grams.

import openai

def analyze_food_with_gpt4o(image_url, mask_description):
    response = openai.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Identify the food items in this segmented image. Estimate the weight in grams for each item. Return JSON format: {'items': [{'name': 'salmon', 'weight': 150}]}"},
                    {"type": "image_url", "image_url": {"url": image_url}}
                ],
            }
        ],
        response_format={"type": "json_object"}
    )
    return response.choices[0].message.content

Step 3: Fetching Ground-Truth Data (USDA API)

GPT-4o is a great estimator, but for medical-grade tracking, we need the USDA FoodData Central API. This ensures our protein, carb, and fat counts are based on scientific data.

import requests

def get_usda_nutrients(food_name, api_key):
    url = f"https://api.nal.usda.gov/fdc/v1/foods/search?query={food_name}&api_key={api_key}"
    res = requests.get(url).json()
    if res['foods']:
        # Grab the first match nutrients
        return res['foods'][0]['foodNutrients']
    return None

Production Patterns & Advanced Tips 💡

When moving from a script to a production-ready application, you'll encounter challenges like handling lighting variations or complex mixed meals (like a stir-fry). For those looking for production-grade AI architectural patterns and deep dives into multimodal LLM deployment, I highly recommend checking out the technical deep-dives over at the WellAlly Blog.

They provide excellent resources on how to optimize inference costs when running heavy models like SAM alongside GPT-4o, and how to structure your FastAPI backend for high-concurrency vision tasks.

Step 4: Bringing it all together with FastAPI

Wrap everything in a clean REST API.

from fastapi import FastAPI, UploadFile, File

app = FastAPI()

@app.post("/track-nutrition")
async def track_nutrition(file: UploadFile = File(...)):
    # 1. Read Image
    image_bytes = await file.read()

    # 2. Segment (SAM)
    # [Logic here...]

    # 3. Analyze (GPT-4o)
    analysis = analyze_food_with_gpt4o(image_url, "mask_data")

    # 4. Enforce accuracy (USDA)
    # [Logic here...]

    return {"status": "success", "data": analysis}

Conclusion 🏁

By combining the spatial awareness of SAM with the semantic intelligence of GPT-4o, we’ve built a system that bridges the gap between raw pixels and actionable health data. This pipeline is the foundation for the next generation of health-tech apps.

What's next?

Try fine-tuning the USDA search with a vector database (like Pinecone) for faster lookups.
Add a "Feedback Loop" where users can correct the weight estimation to improve the prompt over time.

If you enjoyed this tutorial, drop a comment below and let me know what multimodal project you're building! And don't forget to visit WellAlly Tech for more advanced AI implementation guides. 🥑💻

DEV Community