135,692 label words in 49 languages, and not one model call in the path

Munchable reads the ingredients panel on a pack and answers for seven gut conditions. The panel is very often not in English. A jar bought in Lyon says "sirop de glucose-fructose", the same jar in Zagreb says "glukozno-fruktozni sirup", and a scanner that only reads English tells both shoppers it cannot assess their jar.

So the label reader has a table with 135,692 entries in it:

$ node -e "const d=require('./lib/lexicon-generated.json');
> const langs=new Set(Object.keys(d).map(k=>k.split('|')[0]));
> console.log(Object.keys(d).length, langs.size, new Set(Object.values(d)).size)"
135692 193 6424

135,692 language|name keys, resolving to 6,424 canonical ingredient ids. 193 language prefixes appear, though only 49 of them carry 500 entries or more, and those 49 are the ones that matter: a pack sold in the EU prints its list in the languages of the countries it is sold in.

There is no model call anywhere in this path. This post is about why not, and about the four tiers the lookup tries in order.

Why it has to be deterministic

Two reasons, and the second is the one people do not expect.

The obvious one: the rules engine keys on canonical ids, not on free text, so this module is the only bridge between what a pack prints and what a rule can see. A non-deterministic bridge means a verdict that changes between two scans of the same jar.

The less obvious one: contributions are compared. When somebody photographs a label we do not have, the read is matched against other reads of the same product, and agreement is what promotes a row from "somebody's phone said this" to data we will answer from. That comparison is byte equality on the resulting tag list. Put a model in the middle and two identical labels produce two nearly-identical lists that never agree with each other, and the entire consensus mechanism stops working. Same label text in, same tags out, is not a performance choice, it is what makes the crowd usable.

Four tiers, in order

export function resolveGeneratedNameTiered(name: string, lang?: string) {
  const normalized = normalizeToken(name);
  if (!normalized) return null;
  const hand = LEXICON[normalized];
  if (hand) return { id: hand, tier: 'en' };
  const code = lang && /^[a-z]{2}$/.test(lang) ? lang : undefined;
  if (code && code !== 'en') {
    const hit = GENERATED[`${code}|${normalized}`];
    if (hit) return { id: hit, tier: 'lang' };
  }
  const en = GENERATED[`en|${normalized}`];
  if (en) return { id: en, tier: 'en' };
  const any = anyLanguageTable().get(normalized);
  return any ? { id: any, tier: 'any' } : null;
}

1. A hand-written English override table. Small, curated, and deliberately first, because it is where a word that the generated table gets subtly wrong is corrected.
2. The label's own language. If the pack declares French, French is tried before anything else.
3. English, because a lot of labels are bilingual and a lot of ingredient words travel.
4. Any language at all, which is the dangerous one.

The function reports which tier answered, and callers treat the answers differently. That is the whole design: a match is not a match, it is a match with a provenance.

The guard on the fourth tier

An any-language lookup means "some language on earth uses this word for this ingredient", and short words collide across languages spectacularly. So the deterministic layer gates it by length:

const MIN_ANY_LANGUAGE_CHARS = 5;
/** Non-Latin scripts pack more into fewer letters ("سكر"). */
const MIN_ANY_LANGUAGE_CHARS_OTHER_SCRIPT = 3;

Five letters for a Latin-script word, three when the letters are not Latin, because Arabic and Chinese say in three characters what English needs eight for, and a blanket five-character rule would throw away the entire non-Latin half of the table.

Short words are not lost, they just have to earn it. If the same short spelling resolves to exactly one id, in at least two different languages, it is accepted:

const stats = anyLanguageStats(normalizeLexiconName(name));
if (!stats || stats.ids !== 1 || stats.langs < 2 || letters.length < 3) return null;

"Mais" is the example that made this exist. One meaning, several languages, four letters.

There is one more rejection that reads like nonsense until you meet it. A synonym table built from names will happily resolve a bare number to an id, so a stray "5" on a label became an ingredient:

/** Ids like `en:no5` that the synonym table reaches from a bare number; never a resolution. */
const DIGIT_ID = /^en:no\d+$/;

Before the lookup: one fold, shared

Every table in the module, the ingredient names, the allergen words, the advisory splitter, folds text through the same function. That is on purpose. Three tables that each lowercase and strip their own diacritics are three tables that will eventually disagree about what "crème fraîche" looks like, and the bug that produces is an allergen word matching in one place and not another.

The fold also strips percentage quantities, so "milk (3,5%)" is milk, and the footnote marks that labels love.

Tokenising is where the dangerous bug lived

Splitting an ingredients panel looks like text.split(","). It is not, and here is the one that mattered. Take a label ending in:

May contain nuts, peanuts and sesame.

Split on commas first and you get "May contain nuts", which the reader drops as an advisory, and "peanuts and sesame", which it does not recognise as an advisory at all, so it reads them as actual ingredients. A "may contain" had been turned into a "contains", not just for that shopper but for everybody who scanned that product afterwards.

So advisory sentences are cut out whole, before any comma is looked at. Ingredient list headers go too, in every language we read them in, which is a single alternation of "ingredientes | ingredienti | zutaten | ingrediënten | ingredienser | ingredienssit | składniki | sastojci | složení | összetevők | състав | состав" and a handful more.

Then there is prose. Some of what a camera captures is not an ingredient list at all, it is a sentence about the farm. The test for that is grammar rather than length:

const MAX_INGREDIENT_WORDS = 7;

function isProseToken(token: string): boolean {
  const words = token.split(/\s+/).filter(Boolean);
  if (words.length <= MAX_INGREDIENT_WORDS) return false;
  const stops = words.filter((w) => RUN_ON_STOP_WORDS.has(w)).length;
  return stops >= 2;
}

Seven is set just above the longest real ingredient name we have met: "fully refined high oleic sunflower seed oil". Past seven words, a token that resolved to nothing and that the run-on splitter could not take apart needs two or more function words in it to be called a sentence. A sentence is a sentence because it has grammar in it.

There is also a small table of trailing marketing claims in several languages, because "thee van natuurlijke oorsprong" should resolve to tea rather than sit in the backlog as an unscoreable phrase.

A word no language knows stays unknown

The most important rule in the module is what happens on a miss. The token is not dropped, and it is not guessed at. It goes into the tag list as a bare slug with no language prefix, which means the engine counts it as an ingredient it cannot read, and it goes into an explicit unknown list with an honest count beside it.

The slug is produced by an exported function rather than inline, for one specific reason: the curation pass that later works out what that word was has to find the word's slot in the list again, and matching by identical bytes only works if both sides call the same function.

That is what fail closed means here. The reader will tell you it does not know a word. It will never quietly translate it into the nearest thing it does know, because the nearest thing might be the difference between a low FODMAP dinner and an evening somebody planned their week around.

Gaps are not a permanent state, though. Every unknown word goes onto a queue, gets resolved by our own curation, and comes back as a mapping the next scan uses. The point of the deterministic reader is not that it knows everything, it is that it never pretends.

Try it on something in your own kitchen: open app.munchable.app, scan a barcode, and if the product is not on file yet, photograph the ingredients panel. A pack printed in French or Polish is the interesting test. If you would rather read what the engine does with the words once it has them, the ingredient answer pages are one page per ingredient per condition, generated by the same engine, for example does chocolate cause reflux.