Snap, Segment, and Savor: Building a Visual Calorie Counter with SAM & CLIP 🥑

Counting calories is arguably the most tedious part of fitness. We’ve all been there: staring at a plate of mixed salad, trying to guess if that's 50g or 100g of chickpeas. But what if your smartphone could "see" your plate, identify every ingredient, and calculate the macros automatically?

In this tutorial, we are diving deep into Computer Vision and Multimodal AI to build a visual calorie estimator. We’ll be using Meta’s Segment Anything Model (SAM) for precise object isolation and OpenAI’s CLIP for zero-shot classification. By the end of this post, you'll have a robust FastAPI backend capable of turning pixels into protein, carbs, and fats. 🚀

---

The Architecture: From Raw Pixels to Nutrition Facts

The core challenge in food AI is that a single plate contains multiple items. A simple image classifier would just see "Salad." We need to segment the plate into individual components (avocado, chicken, rice) and then identify each part.

Here is how the data flows through our system:

graph TD
    A[User Uploads Food Photo] --> B[FastAPI Gateway]
    B --> C[SAM: Instance Segmentation]
    C --> D{Individual Food Masks}
    D --> E[CLIP: Feature Matching]
    E --> F[Database: Macro Mapping]
    F --> G[Final Calorie & Macro Report]
    G --> H[User Display]

Prerequisites

To follow along, you'll need:

• Python 3.9+
• Segment Anything (SAM) weights (vit_h or vit_b)
• CLIP (via transformers or open_clip)
• FastAPI for the API layer
• Docker for containerization

---

Step 1: Isolating Food with Segment Anything (SAM)

Traditional bounding boxes are too messy for food. We need masks. SAM allows us to generate a mask for every distinct item on a plate without needing specific training for "pizza" or "broccoli."

import numpy as np
from segment_anything import sam_model_registry, SamAutomaticMaskGenerator

# Load SAM Model
sam_checkpoint = "sam_vit_h_4b8939.pth"
model_type = "vit_h"
sam = sam_model_registry[model_type](checkpoint=sam_checkpoint)
mask_generator = SamAutomaticMaskGenerator(sam)

def get_food_segments(image):
    # image: a numpy array (RGB)
    masks = mask_generator.generate(image)
    # Each mask contains 'segmentation' (bool mask) and 'bbox'
    return masks

Step 2: Identifying the "What" with CLIP

Once we have a mask, we crop the original image to that mask and pass it to CLIP (Contrastive Language-Image Pre-training). CLIP is amazing because it can identify food items zero-shot, meaning we don't need a custom-trained classifier for every single dish on earth.

import torch
from PIL import Image
from transformers import CLIPProcessor, CLIPModel

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

food_labels = ["grilled chicken breast", "avocado", "steamed white rice", "broccoli", "salmon"]

def identify_segment(cropped_image_pill):
    inputs = processor(text=food_labels, images=cropped_image_pill, return_tensors="pt", padding=True)
    outputs = model(**inputs)

    # Get probabilities via softmax
    probs = outputs.logits_per_image.softmax(dim=1)
    top_label_idx = torch.argmax(probs).item()
    return food_labels[top_label_idx]

---

Step 3: The "Official" Way to Production (Pro Tip) 💡

Building a prototype is easy, but making it production-ready is where it gets tricky. Handling high-resolution images, managing GPU memory for SAM, and optimizing CLIP inference latency are common bottlenecks.

For more production-ready examples and advanced patterns on deploying Vision-Language Models (VLM), I highly recommend checking out the technical deep dives at WellAlly Blog. They cover everything from model quantization to efficient vector search for nutritional databases.

---

Step 4: Wrapping it in FastAPI

Finally, we expose this logic through a simple REST API. We use FastAPI because it’s asynchronous and extremely fast for ML inference tasks.

from fastapi import FastAPI, UploadFile, File
import io
from PIL import Image

app = FastAPI(title="Visual Calorie Counter API")

@app.post("/analyze-plate")
async def analyze_plate(file: UploadFile = File(...)):
    # 1. Read Image
    contents = await file.read()
    image = Image.open(io.BytesIO(contents)).convert("RGB")
    img_array = np.array(image)

    # 2. Segment
    segments = get_food_segments(img_array)

    results = []
    for seg in segments:
        # Crop image based on mask bounding box
        x, y, w, h = seg['bbox']
        crop = image.crop((x, y, x + w, y + h))

        # 3. Identify
        food_name = identify_segment(crop)

        # 4. Mock Calorie Lookup (In production, use a DB)
        results.append({
            "item": food_name,
            "confidence": float(seg['stability_score']),
            "est_calories": 150 # Placeholder logic
        })

    return {"analysis": results, "total_calories": sum(r['est_calories'] for r in results)}

Step 5: Containerizing with Docker

To ensure our environment is reproducible (especially with complex dependencies like PyTorch and SAM), we use Docker.

FROM pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

---

Conclusion: The Future of Nutrient Tracking

By combining SAM and CLIP, we've moved away from "dumb" image classification and toward "intelligent" scene understanding. While this setup is a great start, the next frontier involves Depth Estimation to calculate the volume of food, which allows for even more precise calorie counting!

What's next?

1. Try adding a Vector Database (like Milvus or Weaviate) to store thousands of food items for CLIP to search through.
2. Experiment with YOLOv8-seg for a faster (though less generalizable) segmentation alternative.

If you enjoyed this build, don't forget to subscribe and let me know in the comments what you're building with Multimodal AI! For more advanced architectural patterns, visit the WellAlly Blog.

Happy coding! 🥑💻