Taming Open Food Facts: building a fast, searchable nutrition database

Dietly — Fri, 17 Jul 2026 16:18:52 +0000

Taming Open Food Facts: building a fast, searchable nutrition database

Every nutrition-app developer hits the same wall. Open Food Facts is a gift — millions of real, barcoded products — but it's crowdsourced: duplicates, unit chaos, half-filled rows and outright impossible values (I've seen a cracker with 6,000 kcal per 100g).

I spent months on the cleanup so you don't have to. The result is DietlyAPI — 4.2M foods behind one fast endpoint (~40ms typical search). This post is about what the cleanup actually involves, because most of it was not obvious to me at the start.

1. Normalization is 80% of the work

Every source has its own idea of a serving. OFF products report per-100g, per-serving, per-package or a mix; fields can be in kJ or kcal, mg or g, sometimes with the unit embedded in a string. Everything gets converted to a canonical per-serving row with grams as the base unit, keeping the original serving description for display.

The nastiest class of bug: values that are individually plausible but jointly impossible. Protein + carbs + fat adding to more than the serving weight. Calories wildly inconsistent with the macro breakdown (4/4/9 rule). Each row gets plausibility checks, and failures either get repaired from other fields (kJ→kcal, per-100g→per-serving) or dropped.

2. Dedupe, then rank by confidence

The same yogurt exists in OFF five times with slightly different barcodes and completion levels. Rather than pick one winner at import time, every row gets a confidence score — data completeness, source reliability, image presence, plausibility-check results — and search ranks by it. The messy duplicates are still there; they just never win.

This turned out to be the single biggest UX improvement: search for "banana" and you get an actual banana with an image, not "BANANA FLAVORED SYRUP CONCENTRATE" from a wholesale distributor.

3. Serving it cheap

The whole thing runs on one 4GB Hetzner box: FastAPI + Postgres (trigram + full-text indexes), Cloudflare in front, 850k pre-rendered static pages for browsing, weekly incremental rebuilds via systemd timers. A public status page publishes real probe latency so "fast" is verifiable, not a claim.

Use it

The free tier needs no card — instant key, 30 req/min:

> curl "https://api.getdietly.com/search?q=greek+yogurt&limit=3"

1. Product page: https://www.getdietly.com/api
2. OpenAPI spec: https://www.getdietly.com/openapi.json
3. Data license: underlying data is ODbL via Open Food Facts — if you show it publicly, credit them (you should anyway; they're the reason any of this is possible). Questions about the pipeline welcome in the comments.

DEV Community: Dietly

Taming Open Food Facts: building a fast, searchable nutrition database