Taming Open Food Facts: building a fast, searchable nutrition database
Every nutrition-app developer hits the same wall. Open Food Facts is a gift — millions of real, barcoded products — but it's crowdsourced: duplicates, unit chaos, half-filled rows and outright impossible values (I've seen a cracker with 6,000 kcal per 100g).
I spent months on the cleanup so you don't have to. The result is DietlyAPI — 4.2M foods behind one fast endpoint (~40ms typical search). This post is about what the cleanup actually involves, because most of it was not obvious to me at the start.
1. Normalization is 80% of the work
Every source has its own idea of a serving. OFF products report per-100g, per-serving, per-package or a mix; fields can be in kJ or kcal, mg or g, sometimes with the unit embedded in a string. Everything gets converted to a canonical per-serving row with grams as the base unit, keeping the original serving description for display.
The nastiest class of bug: values that are individually plausible but jointly impossible. Protein + carbs + fat adding to more than the serving weight. Calories wildly inconsistent with the macro breakdown (4/4/9 rule). Each row gets plausibility checks, and failures either get repaired from other fields (kJ→kcal, per-100g→per-serving) or dropped.
2. Dedupe, then rank by confidence
The same yogurt exists in OFF five times with slightly different barcodes and completion levels. Rather than pick one winner at import time, every row gets a confidence score — data completeness, source reliability, image presence, plausibility-check results — and search ranks by it. The messy duplicates are still there; they just never win.
This turned out to be the single biggest UX improvement: search for "banana" and you get an actual banana with an image, not "BANANA FLAVORED SYRUP CONCENTRATE" from a wholesale distributor.
3. Serving it cheap
The whole thing runs on one 4GB Hetzner box: FastAPI + Postgres (trigram + full-text indexes), Cloudflare in front, 850k pre-rendered static pages for browsing, weekly incremental rebuilds via systemd timers. A public status page publishes real probe latency so "fast" is verifiable, not a claim.
Use it
The free tier needs no card — instant key, 30 req/min:
> curl "https://api.getdietly.com/search?q=greek+yogurt&limit=3"
- 1. Product page: https://www.getdietly.com/api
- 2. OpenAPI spec: https://www.getdietly.com/openapi.json
- 3. Data license: underlying data is ODbL via Open Food Facts — if you show it publicly, credit them (you should anyway; they're the reason any of this is possible). Questions about the pipeline welcome in the comments.
Top comments (0)