Photo In, Macros Out: Designing an AI Meal Analyzer Where the LLM Never Does the Math

#ai #webdev #llm #typescript

"Take a photo of your plate and get calories" is one of the most requested AI features in nutrition apps — and one of the easiest to get wrong. Ask a vision model "how many calories are in this?" and you'll get a confident number that changes each time you ask.

Protealpes has 56 free nutrition tools — calorie, protein, macro, walking and steps-to-calories calculators — plus an AI Meal Analyzer that accepts a photo, a text description, or both. The design rule across all of them: formulas stay visible, and AI doesn't invent the math. Here's the pipeline.

Step 1: The model only identifies items

The AI's job is narrow: split a meal into editable items with estimated amounts. Nothing else. Ask for structured output with a schema so you can validate it:

import { z } from "zod";

const MealItems = z.object({
  items: z.array(z.object({
    name: z.string(),                 // "boiled white rice"
    grams: z.number().positive(),     // estimated portion
    confidence: z.enum(["low", "medium", "high"]),
  })).max(20),
});

type MealItems = z.infer<typeof MealItems>;

async function identify(photo?: Blob, text?: string): Promise<MealItems> {
  const raw = await callVisionModel({
    photo, text,
    instruction: "List the foods and estimated grams. Do not estimate calories or macros.",
  });
  return MealItems.parse(raw); // throws if the model drifts from the schema
}

Validation matters: if the output doesn't match the schema, you retry or fall back to text entry — you don't render it.

Step 2: The user can fix the items

Portion estimates from photos are the weakest link, so every item is editable before anything is calculated. "250 g" of rice might really be 180 g — the user knows, the model doesn't.

Step 3: Deterministic lookup against a food table

Nutrition comes from a USDA-derived table, per 100 g:

interface FoodRow { id: string; names: string[]; kcal: number; protein: number; carbs: number; fat: number }

function match(name: string, table: FoodRow[]) {
  const n = name.toLowerCase();
  return table.find((r) => r.names.some((alias) => n.includes(alias)));
}

export function analyze(items: MealItems["items"], table: FoodRow[]) {
  return items.map((it) => {
    const row = match(it.name, table);
    if (!row) return { ...it, estimated: true, kcal: null };   // unmatched stays marked
    const f = it.grams / 100;
    return {
      ...it, estimated: false,
      kcal: Math.round(row.kcal * f),
      protein: +(row.protein * f).toFixed(1),
      carbs: +(row.carbs * f).toFixed(1),
      fat: +(row.fat * f).toFixed(1),
    };
  });
}

Unmatched rows stay clearly marked as estimated instead of being silently filled in.

Step 4: Cooked vs raw is a different food

A classic tracking error: 250 g of boiled white rice is roughly 325 kcal, while dry rice is around 365 kcal per 100 g. They're separate rows in the table, and the matcher prefers the cooked row unless the item says "dry" or "uncooked". The same applies to pasta, oats and legumes.

Step 5: Calculators stay formula-first

The non-AI tools use published formulas with visible math: Mifflin-St Jeor for BMR, an activity factor for TDEE, 4/4/9 kcal per gram for macros, and explicit g/kg bands for protein (e.g. 0.8–1.2 for general health, 1.6–2.2 for muscle gain). Metric and US units are both supported, converted at the edges so the core math only deals with kg and cm.

Takeaways

Use the LLM for perception and parsing, not arithmetic.
Enforce a schema on model output and validate it.
Let users edit portions before calculating.
Mark unmatched items as estimated; never hide uncertainty.

Try the tools at protealpes.net. Where do you draw the line between model output and deterministic code in your AI features?

DEV Community