We deleted 45% of our stored ingredient tags, and used our own engine as the oracle

Munchable scans a packaged food's barcode and tells you whether it fits your gut condition. You can see the conditions it reasons about at munchable.app/conditions, and 373 worked examples of the ingredient reasoning at munchable.app/answers.

The catalogue behind it is a single Postgres table with a million-odd product rows. Each row with a readable label carries an array of ingredient tags: en:wheat-flour, en:sugar, en:cheese, in the order the pack prints them. That array is the most expensive column we own. It rides in every lookup the phone makes and gets cached on the device, and the database sits on a half gigabyte quota.

Measuring it turned up something awkward. Forty-five percent of the stored tags were tags our own engine computes.

The duplication

The engine keeps a parent graph over ingredients. en:carrot has en:vegetable as a parent, en:cheddar leads up to en:cheese and then to en:dairy. Normalising a product walks the printed tags and expands every ancestor, because a rule that speaks about dairy has to fire on a pack that printed cheddar.

Our seeded rows had the ancestors written into the array as well. So a row that printed cheddar stored cheddar, cheese and dairy, and at scan time the engine derived cheese and dairy from cheddar a second time, got the same answer, and threw it away.

Thirty-eight megabytes of a hundred and forty-eight, for a value that is recomputed on every read anyway. Derived data in a row is a cache, and the bill for this one was being paid in quota and in mobile bandwidth while buying nothing, because the deriver is deterministic and already runs on the request path.

So: delete them. The interesting part of the job is not the delete, it is proving the delete changes nothing.

Three rules about what may go

It has to be an ancestor of an earlier tag, not merely of some tag. Label order is meaningful. The engine ranks an ingredient by where it appears, and the first thing that normalisation does with the list is walk it front to back. A generic tag that the seed placed before its child is treated as printed, gets its own rank, and may be reported in its own right. One that appears after a tag that already implies it is the pure duplicate. Same word, two different meanings, decided entirely by position:

function compact(tags: readonly string[]): string[] {
  const implied = new Set<string>();
  const out: string[] = [];
  for (const t of tags) {
    if (PROTECTED.has(t) || !implied.has(t)) out.push(t);
    for (const a of ancestors(t)) implied.add(a);
  }
  return out;
}

The graph used for the walk has to be the one every build agrees on. Our ingredient graph has three layers: the seed, hand-curated extensions, and an AI-proposed overlay that gets published separately and that a given phone may hold an older or a newer copy of. Compacting against a layer a client might not have is how you turn storage into a version-dependent answer. So the walk uses the seed plus the hand layer only, which are the layers that win every merge and are therefore present in every build.

Allergen tags never go, generic or not. That is PROTECTED above. Allergen notices are the one place in the app where a warning may only ever be added, never cleared, and the allergen check reads what the pack printed rather than what the graph implies. A generic word like en:nut counts when the pack printed it. Dropping one because the graph says it is redundant would be a quota saving that silently changes a safety-relevant output, which is the worst trade in the product.

Use the engine as the oracle

Rules you reasoned about are rules you can get wrong. A million rows is too many to eyeball and exactly the right number for differential testing, so every candidate row is run through everything the engine can say, twice, and is only written if the two agree byte for byte:

const PROFILE: Profile = {
  conditions: ALL_CONDITIONS,          // every condition at once
  allergens: [...ALLERGEN_IDS],        // every allergen declared
  healthy: [...HEALTH_PREFERENCE_IDS], // every preference switched on
};

function everything(row: CatalogProduct): string {
  const product = shapeProductRow(row);
  const normalized = normalizeProduct(product);
  return JSON.stringify({
    fit: fitCheck(product, PROFILE),
    allergens: checkAllergens(normalized, PROFILE.allergens ?? []),
    healthy: checkHealthy(normalized, PROFILE.healthy ?? []),
  });
}

if (everything(row) !== everything({ ...row, ingredientsTags: next })) {
  differs++;            // leave the row exactly as it is, and report it
  continue;
}

The maximal profile is the point. A user has one condition or two; the oracle asks the question for all of them simultaneously, with every allergen and every preference on, so any difference anywhere in the output space shows up. A row that differs is not debugged in the moment and not forced through. It is left alone and printed, and the first twenty get listed at the end of the run. Those are the rows where my three rules above were wrong, and the script's job is to surface them, not to be right about them.

The Postgres end of it

Keyset pagination on the primary key, 2,000 rows a page, and a single batched update per page with the old value as a guard:

update catalog.products as p
   set ingredients_tags = v.tags
  from jsonb_to_recordset($1::jsonb) as v(barcode text, old text[], tags text[])
 where p.barcode = v.barcode
   and p.source = 'seed'
   and p.ingredients_tags = v.old

Comparing against v.old is optimistic concurrency for a long-running job. The script reads a page, thinks for a while, and writes. If a user photographed a label in that window and the row changed, the predicate fails, the row is skipped rather than overwritten, and the count of affected rows tells us it happened. A migration that cannot tolerate concurrent writes is a migration that needs a maintenance window, and we would rather not have one.

Two smaller things worth knowing:

• The batch is passed as the array itself, not JSON.stringify(batch). The postgres driver serialises a jsonb parameter for you, so a pre-stringified value arrives as a JSON string and jsonb_to_recordset gets a scalar.
• updated_at is deliberately not bumped. This changes how a row is stored, not what the product is, and the device cache keys off that column. Bumping it would have invalidated every cached product on every phone to deliver a result the phone cannot observe.

Then the part that actually returns the disk:

vacuum full catalog.products;

An UPDATE in Postgres writes a new row version and leaves the old one as a dead tuple. A plain VACUUM makes that space reusable by the table. Neither hands anything back to a hosting quota. VACUUM FULL rewrites the table with only its live rows, which does give the space back, and it needs free disk for the compacted size rather than the current one. That is the detail that made it possible here: on a nearly full disk, the rewrite of a smaller table fits where a copy of the bigger one would not. It takes an exclusive lock for about a minute, and a dry run that tells you how many rows will change is how you decide whether the lock is worth taking.

The script runs in dry-run mode by default, prints before and after sizes from pg_total_relation_size, and needs --apply to write anything. For a one-off data deletion against the only copy of your production catalogue, that default is not ceremony.

See it in the open

The ancestor expansion this all rests on is what makes the public answer pages work, and they are the easiest way to watch it happen. Is cheddar high in lactose? and Is cheese high in lactose? are two pages where one answer is reached through the other's rule. Is onion low FODMAP? shows the same for the wordings a label uses for one ingredient.

To see it on a real pack, with the compacted row coming off the wire, sign in at app.munchable.app and scan something in the cupboard.