DEV Community

Cover image for Teaching a Dictionary to Learn Instead of Hand-Curating It Forever
Suyash Vashishtha
Suyash Vashishtha

Posted on Originally published at myvitals.co.in

Teaching a Dictionary to Learn Instead of Hand-Curating It Forever

"Fasting Blood Sugar (FBS)" in mg/dL from Lab A and "Glucose, Fasting" in mmol/L from Lab B, a year apart, are the same measurement. If your app can't figure that out, it can't draw a trend line — which is the entire point of MyVitals.

Four tiers, cheapest first, first match wins

  1. Exact alias match — normalize the raw name, compare against known aliases. Instant.
  2. Exact match on the extraction model's own guess (canonicalNameGuess) — added after a real bug, below.
  3. Fuzzy similarity (Dice coefficient + token-set comparison) with a hard guard against matching "HDL" to "non-HDL Cholesterol" just because they're close.
  4. LLM fallback — batched, retried, timeboxed.
  5. Auto-create — mint a new entry. Report still saves. Dictionary just grew instead of getting smarter.

Whatever tier wins, the raw name gets appended as a new alias. The dictionary self-improves across every user — the fifth report where a lab prints "PCV" for hematocrit, tier 1 catches it instantly. That's the real moat: not "we can read a PDF," but "we get cheaper and more accurate with every report anyone uploads."

The bug that made two Hematocrits

The model's canonicalNameGuess used to be used only to name a new entry, never to find an existing one. Lab A prints "HCT," Lab B prints "PCV / Haematocrit" — neither raw name matches, fuzzy misses, both fall through to auto-create, and auto-create names both of them "Hematocrit" using the guess. Now you have hematocrit and hematocrit_1, splitting one measurement into two trend lines depending on which lab drew your blood — while the model had correctly identified both as Hematocrit the whole time. The pipeline just never asked.

Tier 2 (checking the guess against the dictionary before auto-create) fixed it — deliberately exact-only, never fuzzy, because a wrong merge puts two different analytes on one chart with zero indication it happened. That's worse than an extra row. Cleanup merged 14 duplicate groups, 126 → 112 metrics, full backup first, refused to run if any reading would come out worse.

Units lie in more creative ways than names do

µg/dL (micro sign) and μg/dL (Greek mu) and ug/dL (gave up on Unicode) are one unit in three costumes. The name-matching normalizer strips non-alphanumeric characters, collapsing all three into different strings that then compare as different units — 15 of 27 "unconvertible" readings turned out to be exactly this: correctly matched data, silently excluded from its own chart over a font rendering difference.

Fix: a second normalizer, used only for unit comparison, that knows µ/μ/u are the same prefix — and nothing else. It does not get to decide mg/dL ≈ mmol/L, or that μU/mL ≈ μIU/mL (one character apart, not the same unit). Any real equivalence claim needs a human to assert it explicitly. Correctness beats completeness when there's a chart on the other end.


👉 Try MyVitals now — upload an old report and watch it reconcile into a trend line.

Top comments (0)