Why fuzzy search alone produces bad food results

String similarity can rescue a typo, but it cannot tell a useful nutrition record from an empty duplicate. A search for greek yogrt should tolerate the missing letter. It should also prefer a complete Greek-yogurt record over a product whose name happens to match perfectly but whose calories and macros are missing.

Similarity solves only the spelling problem

Trigram similarity is a practical first pass for food names. It breaks text into overlapping fragments, so yogrt remains close to yogurt. An indexed similarity operator can find candidates without scanning millions of rows.

SELECT name, similarity(lower(name), lower(:query)) AS sim
FROM food_names
WHERE lower(name) % lower(:query)
ORDER BY sim DESC
LIMIT 60;

That query answers “which names look alike?” It does not answer “which result should a calorie tracker show first?” Those are different questions.

Food queries mix products, brands and descriptions

People search for nutela, kroger cheddar, protein yogurt vanilla and barcode-like strings. Sometimes the brand is in a separate field; sometimes it is embedded in the product name; sometimes the useful word appears only in a translated label. A production search therefore needs candidate generation across both name and brand, followed by one ranking pass.

Duplicates are normal, not exceptional

Community datasets accumulate the same product through packaging updates, translations, regional barcodes and repeated submissions. Deleting every near-duplicate is risky because two similar names can represent different sizes or recipes. Returning all of them is equally bad.

Dietly collapses exact candidate IDs and then ranks deterministic winners. For a barcode shared by multiple source rows, it prefers an image, then higher confidence, then a stable ID tie-break. The important design principle is repeatability: the same request should not reshuffle equivalent records between page loads.

Sparse records should not win on wording alone

A perfect name match with no calories is usually less useful than a 0.91 similarity match with calories, protein, fat and carbohydrate. Ranking can encode that preference without pretending the data is scientifically verified.

Signal	What it helps with
Exact name match	Obvious intent
Name/brand similarity	Typos and partial wording
Image present	Recognition in scan/search UI
Core macros present	Usability in trackers
Plausibility checks	Obvious unit or entry mistakes
Completeness	Choosing between similar records
Confidence	A final quality signal, not a truth probability

ORDER BY
  is_exact_match DESC,
  has_image DESC,
  core_macros_complete DESC,
  values_plausible DESC,
  text_similarity DESC,
  completeness DESC,
  confidence DESC;

Confidence must remain explainable

A confidence score should mean “this record passed more of our internal quality signals,” not “there is a 93% chance the label is correct.” Useful inputs include source type, nutrient completeness, unit consistency and plausible ranges. Keep the raw source and nullable fields so clients can make their own decisions.

Test ranking, not just matching

Create a frozen query set containing misspellings, brands, generic foods, multilingual names and deliberately sparse duplicates. Assert that the expected useful record appears in the first few results. Precision at rank one matters more to an autocomplete user than whether the correct product exists somewhere among 200 candidates.

Candidate generation and ranking should be separate stages

Trying to express every rule in one enormous SQL score makes the search difficult to tune. A cleaner architecture first retrieves a generous but bounded candidate set, then applies business ranking. Exact name and prefix matches can form one candidate stream; fuzzy name matches and fuzzy brand matches can form others. Merge them by stable food ID before applying completeness and quality signals. This keeps typo recovery from displacing obvious exact matches and prevents the same product from occupying several result slots.

The candidate limit is an engineering control. Too small, and a useful record never reaches the ranker. Too large, and every keystroke performs unnecessary joins and sorting. Measure recall on a frozen query set while changing that limit. The correct number depends on the selectivity of the index, the languages in the catalog and how much ranking work happens outside PostgreSQL.

Query intent changes what “best” means

A generic query such as banana usually benefits from a common complete food near the top. A specific query such as acme banana yogurt 150g should reward brand and package wording more heavily. Numeric tokens can indicate size, fat percentage or a barcode fragment. Removing every number as noise can collapse distinct products; treating every number as decisive can overfit messy labels.

One practical approach is to calculate several interpretable features instead of one opaque similarity value: exact normalized name, prefix match, brand-token overlap, trigram similarity, token coverage and whether meaningful numeric tokens agree. Log those features for sampled searches. When a result looks wrong, you can explain which signal won and adjust it without guessing.

Autocomplete needs product rules as well as search rules

Debounce input, require at least two meaningful characters and cancel stale requests when the user continues typing. Otherwise a slow response for gre can arrive after the response for greek yogurt and replace better results. Cache popular normalized queries briefly, but include filters and locale in the cache key. Never mix results for different sources or categories because two requests happened to share the same text.

The UI should reveal enough context to distinguish candidates: product name, brand, image and a consistent nutrient such as calories per 100 g. Do not show confidence as a mysterious percentage. It is more useful internally for ranking than as a consumer-facing promise. Give users a way to report or bypass a bad match; behavioral feedback exposes ranking failures that offline test sets miss.

Measure usefulness, not just database speed

Track median and tail latency, but also measure empty-result rate, reformulation rate, selection position and the share of searches where the user immediately returns. A ten-millisecond query that consistently puts an unusable duplicate first is not a successful search. Review results by country, language and query length because aggregate metrics can hide a catalog that works well for English supermarket brands and poorly everywhere else.

Finally, keep a manual evaluation set with the expected intent and acceptable result IDs. Run it whenever ingestion, indexes or ranking weights change. Food data evolves daily, so exact snapshots will move; the test should protect usefulness and invariants rather than freeze the catalog forever.

Try the behavior: DietlyAPI searches 4.7M+ indexed foods with fuzzy name and brand matching plus confidence-aware ranking. Public reads can be tested before signup at the API page.

---

Originally published at getdietly.com. Data from the Dietly Nutrition API — 4.7M+ indexed foods, free tier available.