45% of our stored ingredient tags were derivable, so we deleted them

Munchable's product rows carry an array of ingredient tags: en:water, en:sugar, en:carrot, the canonical id for each thing on the label. The rules engine reads that array and decides whether the product suits your gut.

The arrays arrived pre-expanded. A row with en:carrot on it also carried en:vegetable, en:root-vegetable and whatever else sits above carrot in the taxonomy. Reasonable at import time, and it has the nice property that a rule keyed on a broad category matches without anybody walking a graph at query time.

Except we walk the graph anyway. normalizeProduct expands ancestors itself, every time, on every lookup, because it has to: the engine cannot trust that a row was expanded against the same version of the taxonomy the current build holds. So every stored ancestor was being re-derived from the child sitting next to it in the same array.

I measured it before touching anything:

• 45% of all stored tags were ancestors of another tag in the same row.
• 38 MB of the 148 MB that rows with ingredients occupy.
• And every one of them rides in every lookup payload a phone downloads and caches.

The last point is the one that mattered most. This is a mobile app whose product cache lives on the device. Redundant tags are not only disk on a database with a hard quota, they are bytes over a shop's wifi and bytes in the phone's storage, paid for on every single product anyone ever scans.

The rule is three words longer than you expect

Drop a tag if it is an ancestor of an earlier tag in the same row.

function compact(tags: readonly string[]): string[] {
  const implied = new Set<string>();
  const out: string[] = [];
  for (const t of tags) {
    if (PROTECTED.has(t) || !implied.has(t)) out.push(t);
    for (const a of ancestors(t)) implied.add(a);
  }
  return out;
}

"Earlier", not "anywhere in the row". That distinction is the whole correctness argument, and it took a reading of the normalizer to find.

normalizeProduct walks the tag list in order. By the time it reaches a tag that an earlier tag already implied, it has added that ancestor as derived and skips it without touching a rank or a list. Removing it from storage therefore changes nothing: the engine lands in exactly the same place.

An ancestor that appears before its child is a different animal. The engine has not derived it yet, so it treats it as printed on the label, gives it its own rank in the ingredient order, and may report it as unscored. Drop that one and you have changed what the engine believes the label said. So it stays, redundant or not.

Ordering was load bearing in a data structure I had been thinking of as a set. It was not a set. It was the label, in order.

Which graph you walk is a correctness question

Our taxonomy has three sources: the imported seed, hand-curated extensions, and an AI curation overlay that fills gaps. The compaction walks the first two and deliberately ignores the third:

const GRAPH = { ...TAXONOMY_PARENTS, ...TAXONOMY_EXTENSIONS };

The overlay is versioned and devices hold different copies of it. A tag dropped because today's overlay says it is derivable would fail to be re-derived on a phone holding last week's overlay, and the deletion is permanent. Seed and hand entries win every merge in the engine's taxonomy resolution, so what this walk derives, every build derives, on every device, forever.

That is the shape of the rule for anything that deletes derivable data: you may only delete what the least informed consumer can rebuild.

One category of tag is never dropped

const PROTECTED = new Set<string>();
for (const sources of Object.values(ALLERGEN_SOURCES))
  for (const s of sources) PROTECTED.add(s.tag);

Allergen source tags stay, generic or not, whatever the graph says.

The allergen layer reads printedTags rather than the expanded list, because a generic word such as en:nut counts as a warning only when the pack actually printed it. Removing one because a more specific nut appears earlier would be textbook-correct against the taxonomy and would silently remove an allergen notice.

The safety-critical layer does not get a clever optimisation. It gets an exemption, and the exemption makes its output byte-identical by construction rather than by argument.

Every row is re-run before it is written

The argument above is good. It is not good enough to run a destructive UPDATE over a million rows on the strength of, so the script proves it per row:

const PROFILE: Profile = {
  conditions: ['gerd', 'ibs', 'lactose', 'bam', 'gastroparesis', 'sibo', 'ibd'],
  allergens: [...ALLERGEN_IDS],
  healthy: [...HEALTH_PREFERENCE_IDS],
};

function everything(row: CatalogProduct): string {
  const product = shapeProductRow(row);
  const normalized = normalizeProduct(product);
  return JSON.stringify({
    fit: fitCheck(product, PROFILE),
    allergens: checkAllergens(normalized, PROFILE.allergens),
    healthy: checkHealthy(normalized, PROFILE.healthy),
  });
}

Every condition, every allergen and every healthy preference switched on at once, so the comparison covers everything the engine is capable of saying about a product rather than everything one profile would ask. Run it against the old tags, run it against the new tags, compare the strings. A row whose output differs in a single byte is left exactly as it is and reported by barcode.

This is the part I would keep if I kept nothing else from the script. The test is not "is my reasoning about the taxonomy sound", it is "does the output change", asked of every row individually. Reasoning about a graph is where you make mistakes. Comparing two JSON strings is where you catch them.

The rows are walked keyset-paged on the primary key, 2,000 at a time, so nothing holds a long transaction.

Two small things that are easy to get wrong

updated_at is not bumped. This changes how a row is stored, not what the product is. Bumping it would have made a million products look freshly edited to every cache, sync check and "recently changed" query in the system, which is a fine way to turn a storage optimisation into a thundering herd.

Label order is preserved for the tags that stay. The array is the ingredient list in the order the pack prints it, and the engine ranks by position. Compaction filters; it never sorts.

VACUUM FULL is the step that actually gives you the disk

pnpm exec tsx scripts/db-compact-tags.mts            # dry run: verify every row, report
pnpm exec tsx scripts/db-compact-tags.mts --apply    # rewrite verified rows, then VACUUM FULL

An UPDATE in Postgres writes a new row version and leaves the old one behind until vacuum. A plain VACUUM marks that space reusable by the table. Neither returns anything to the quota. VACUUM FULL rewrites the table with only its live rows and hands the disk back, at the cost of an exclusive lock for a minute or so on a table that is otherwise read-only in the request path.

If you are on a plan with a hard size limit, the --apply run that skips the vacuum looks like it did nothing at all, and you will spend an afternoon working out why.

Dry run is the default and prints the full report without writing. For a script whose job is to delete data from every row of the main table, the flag should be on the dangerous path, never on the safe one.

The output, on the web

The engine that all of this was verified against is the same one behind our public pages, so you can read the result of those tag arrays without an app:

• Is onion low FODMAP? is one ingredient, one condition, engine output rendered as a page.
• The answer index is every ingredient and condition pair we hold a rule for.
• The low FODMAP guide is the condition that page sits under.

And the app that downloads the newly smaller payloads is at munchable.app. It is about 38 MB lighter across the catalogue, which nobody will ever notice, which is exactly the right outcome.