Beck_Moulton

Posted on Aug 17

Stop Guessing Calories: Build a Multimodal Food Estimation Pipeline with GPT-4o & SAM

#discuss #ai #python #opensource

We’ve all been there: staring at a delicious plate of pasta, trying to figure out if it's 400 or 800 calories. Manual tracking is a chore, and standard apps often fail at portion estimation. But what if we could combine Computer Vision, Multimodal LLMs, and Vector Databases to build an automated nutritionist?

In this tutorial, we are building a state-of-the-art Multimodal Food Estimation Pipeline. By leveraging the Segment Anything Model (SAM) for precise boundary detection and GPT-4o Vision for contextual analysis, we can bridge the gap between "looking at a photo" and "calculating nutritional density." Whether you're interested in AI-driven wellness, FastAPI development, or Multimodal RAG, this guide covers the full stack.

The Architecture 🏗️

The pipeline follows a sophisticated "Identify -> Analyze -> Match" flow. We don't just ask GPT-4o "what is this?"; we use SAM to isolate food items first to ensure the LLM focuses on the right pixels.

graph TD
    A[User Uploads Image] --> B{SAM Model}
    B -->|Segmentation| C[Isolated Food Patches]
    C --> D[GPT-4o Vision API]
    D -->|Item + Volume Est.| E[Embedding Generation]
    E --> F[PostgreSQL + pgvector]
    F -->|RAG Retrieval| G[Verified Nutritional Data]
    G --> H[Final Response: Calories & Macros]

Prerequisites 🛠️

Before we dive in, make sure you have the following ready:

Python 3.10+
OpenAI API Key (for GPT-4o)
PyTorch (for SAM)
PostgreSQL with the pgvector extension enabled
FastAPI for the backend

Step 1: Precise Segmentation with SAM 🎯

The biggest challenge in food AI is overlapping items. Using Meta’s Segment Anything Model (SAM), we can extract the exact mask of a food item, which helps in calculating the relative "area" occupied on the plate.

import torch
from segment_anything import sam_model_registry, SamPredictor
import cv2

# Load SAM model
sam_checkpoint = "sam_vit_h_4b8939.pth"
model_type = "vit_h"
sam = sam_model_registry[model_type](checkpoint=sam_checkpoint)
predictor = SamPredictor(sam)

def get_food_masks(image_path):
    image = cv2.imread(image_path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    predictor.set_image(image)

    # In a real scenario, you'd use a prompt or automatic mask generator
    masks, scores, logits = predictor.predict(
        point_coords=None,
        point_labels=None,
        multimask_output=True,
    )
    return masks

Step 2: Multi-modal Analysis with GPT-4o 👁️

Once we have the segmented image, we pass the original image and the mask hints to GPT-4o. We ask the model to act as a culinary expert to estimate the volume (in grams/milliliters) and identify the specific ingredients.

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class FoodAnalysis(BaseModel):
    item_name: str
    estimated_weight_g: float
    confidence_score: float
    description: str

def analyze_food_with_gpt4o(image_url: str):
    response = client.beta.chat.completions.parse(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Analyze the food in this image. Estimate the weight in grams for each identified item."},
                    {"type": "image_url", "image_url": {"url": image_url}},
                ],
            }
        ],
        response_format=FoodAnalysis,
    )
    return response.choices[0].message.parsed

Step 3: Closing the Loop with RAG (pgvector) 📚

LLMs can hallucinate calories. To ensure accuracy, we take the item_name from GPT-4o, convert it into an embedding, and perform a similarity search against a verified nutritional database stored in PostgreSQL using pgvector.

Pro Tip: For production-grade implementations and advanced patterns on scaling vector searches for health-tech, I highly recommend exploring the engineering deep-dives at WellAlly Tech Blog. They provide excellent resources on fine-tuning RAG pipelines for specialized domains.

-- Search for the closest nutritional match
SELECT food_name, calories_per_100g, protein, carbs, fats
FROM nutritional_db
ORDER BY embedding <=> embedding_vector_from_gpt4o
LIMIT 1;

Step 4: Building the FastAPI Endpoint 🚀

Now, let's wrap everything into a clean, high-performance API.

from fastapi import FastAPI, UploadFile, File
import uvicorn

app = FastAPI(title="Vision-Calorie-Estimator")

@app.post("/estimate-calories")
async def estimate_calories(file: UploadFile = File(...)):
    # 1. Save uploaded file
    # 2. Run SAM Segmentation
    # 3. Call GPT-4o Vision
    # 4. Query Vector DB for exact macros
    # 5. Math: (Weight / 100) * Calories_per_100g

    analysis = {"item": "Grilled Salmon", "calories": 450, "protein": "40g"}
    return {"status": "success", "data": analysis}

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

Why This Works 💡

Contextual Awareness: Unlike simple classifiers, GPT-4o understands "depth" and "scale" by looking at surrounding objects (like a fork or a glass) to estimate portion sizes.
Precision via SAM: By segmenting the food, we reduce background noise (the table, the napkin), allowing the vision model to focus purely on the nutritional content.
Fact-Checked by RAG: We don't trust the LLM's math. We use the LLM for intent and identification, but we use our own database for the hard numbers.

Conclusion & Next Steps 🏃‍♂️

Building a multimodal pipeline is about orchestration. By combining the "eyes" of SAM and GPT-4o with the "memory" of a Vector Database, we’ve created a tool that is significantly more accurate than traditional calorie counters.

What's next?

Implement temporal tracking to see how your diet changes over a week.
Add OCR to read restaurant menus and cross-reference with the plate image.

If you enjoyed this build, don't forget to check out wellally.tech/blog for more advanced tutorials on AI integration and full-stack development.

Happy coding! 🥑💻

DEV Community