DEV Community

Beck_Moulton
Beck_Moulton

Posted on

Snap & Eat: Building a Real-time Calorie Tracker with GPT-4o and YOLOv10

#ai #machinelearning #python #react

Counting calories is, quite frankly, a full-time job. We’ve all been there: staring at a bowl of ramen, manually searching for "tonkotsu broth" in an app, and guessing if it's 300g or 500g. But what if your phone could just see the food and calculate the macros for you?

In this tutorial, we are diving deep into AI-powered nutrition and Computer Vision to build a real-time dietary analysis system. By combining the blazing-fast detection of YOLOv10 with the multimodal reasoning of GPT-4o, we can transform raw pixels into actionable nutritional macro data. Whether you're interested in GPT-4o Vision API integration or real-time object detection, this guide covers the full stack from mobile to cloud.

For those looking for even more production-ready patterns and advanced AI architecture deep-dives, definitely check out the comprehensive resources at WellAlly Tech Blog.

🏗 The Architecture: From Pixels to Protein

To achieve real-time performance without draining a smartphone battery, we use a hybrid approach. YOLOv10 handles the lightning-fast object localization (finding the food on the plate), and GPT-4o handles the complex "reasoning" (estimating volume and nutritional density).

graph TD
    A[React Native App] -->|Video Stream/Snap| B(FastAPI Backend)
    B --> C{YOLOv10 Detector}
    C -->|Bounding Box/Crop| D[GPT-4o Vision API]
    D -->|Multimodal Analysis| E[Nutritional Mapping]
    E -->|Structured JSON| B
    B -->|Macro Data: Cal/P/C/F| A
    style D fill:#f9f,stroke:#333,stroke-width:2px
Enter fullscreen mode Exit fullscreen mode

🛠 The Tech Stack

  • Frontend: React Native (Mobile capture)
  • Object Detection: YOLOv10 (SOTA for real-time inference)
  • Multimodal Logic: OpenAI GPT-4o
  • Backend: FastAPI (Python)
  • Validation: Pydantic

Step 1: Real-Time Detection with YOLOv10

We use YOLOv10 because it eliminates the need for Non-Maximum Suppression (NMS), making it incredibly efficient for edge-to-cloud pipelines. Our backend receives an image and identifies exactly where the food is.

from ultralytics import YOLOv10
import cv2

# Load the pre-trained food detection model
model = YOLOv10('yolov10n_food.pt')

def detect_food(image_path):
    results = model(image_path)
    # Extract the bounding boxes for GPT-4o to focus on
    for result in results:
        boxes = result.boxes.xyxy.tolist()
        return boxes # Returning coordinates for cropping
Enter fullscreen mode Exit fullscreen mode

Step 2: The "Brain" – GPT-4o Vision Analysis

Once we have the localized food, we send the crop (or the full image with markers) to GPT-4o. The magic here lies in the System Prompt. We need structured data (JSON), not a poetic description of a burger.

import openai
from pydantic import BaseModel

class NutritionData(BaseModel):
    food_name: str
    estimated_weight_g: int
    calories: int
    protein_g: float
    carbs_g: float
    fats_g: float

def analyze_nutrition(image_url):
    response = client.beta.chat.completions.parse(
        model="gpt-4o",
        messages=[
            {
                "role": "system", 
                "content": "You are a professional nutritionist. Estimate the weight and nutritional content of the food in the image."
            },
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Analyze this meal."},
                    {"type": "image_url", "image_url": {"url": image_url}}
                ]
            }
        ],
        response_format=NutritionData,
    )
    return response.choices[0].message.parsed
Enter fullscreen mode Exit fullscreen mode

Step 3: Fast API Integration

We wrap everything in a FastAPI endpoint to bridge our React Native frontend with our AI logic.

from fastapi import FastAPI, UploadFile, File

app = FastAPI()

@app.post("/analyze-plate")
async def analyze_plate(file: UploadFile = File(...)):
    # 1. Save uploaded file
    # 2. Run YOLOv10 detection
    # 3. Call GPT-4o Vision API
    # 4. Return the structured macros
    nutrition_info = analyze_nutrition(image_link)
    return {"status": "success", "data": nutrition_info}
Enter fullscreen mode Exit fullscreen mode

💡 The "Official" Way to Scale

While this implementation is great for a prototype, scaling a multimodal AI system requires robust observability and prompt versioning. I've learned a ton about optimizing these types of pipelines by following the engineering standards over at WellAlly Tech Blog. They have fantastic articles on handling high-throughput FastAPI applications and managing LLM costs—critical if you're planning to move beyond a hobby project!

📱 Frontend: React Native Snippet

On the mobile side, we use react-native-vision-camera to capture the frame and push it to our /analyze-plate endpoint.

const takePicture = async () => {
  const photo = await camera.current.takePhoto();
  const formData = new FormData();
  formData.append('file', {
    uri: photo.path,
    type: 'image/jpeg',
    name: 'meal.jpg',
  });

  const response = await fetch('https://your-api.com/analyze-plate', {
    method: 'POST',
    body: formData,
  });
  const macros = await response.json();
  console.log(`Calories: ${macros.data.calories}kcal 🚀`);
};
Enter fullscreen mode Exit fullscreen mode

🎯 Conclusion

We’ve just built a bridge between the physical world (pixels) and structured health data (macros). By leveraging YOLOv10 for the "where" and GPT-4o for the "what," we create a seamless user experience that makes health tracking as easy as taking a selfie.

What's next?

  • Implementing a "History" feature with PostgreSQL.
  • Fine-tuning YOLOv10 on a custom dataset of regional cuisines.
  • Adding a chatbot to give advice based on the day's total macros.

What do you think? Is vision-based calorie tracking the future of fitness, or are we still too reliant on AI "estimation"? Let me know in the comments! 👇

Top comments (0)