Product Matching Across Retailers: Identifiers, Fuzzy Titles and Confidence Scores

Product matching is the process of deciding whether two listings from different retailers refer to exactly the same product, so that their prices, stock and ratings can be compared like for like. It sounds trivial until you try it: one retailer calls it "Acme Wireless Earbuds Pro, Black", another "Wireless Earbuds Pro (Black) by Acme", and a third lists the Lite model with an almost identical title.

Every comparison built on top, from price positioning to assortment gaps, inherits the quality of the match. A wrong match produces a confident, wrong answer. This post builds a layered matcher: identifiers first, then hard attribute conflicts, then fuzzy titles, with a confidence score that decides what is matched automatically and what goes to a person. All of the code was run against sample listings.

The approach in one list

1. Validated identifiers. If both listings carry a valid GTIN, the GTIN decides.
2. Hard conflicts. Different brand, size, pack count, colour or model word means no match, however similar the titles look.
3. Fuzzy title similarity. For what is left, score how alike the titles are.
4. Confidence thresholds. High scores match automatically, middling scores go to review, low scores are rejected.
5. Measurement. Check precision and recall against hand-labelled pairs, and tune the thresholds.

Step 1: validate identifiers before trusting them

GTINs (the numbers behind barcodes) are the best matching key there is, but only if they are real. Scraped GTINs arrive as 8, 12, 13 or 14 digits, sometimes with spaces, and sometimes simply wrong. Normalise them all to 14 digits and check the final digit, which is calculated from the others:

def normalise_gtin(raw: str | None) -> str | None:
    """Return a valid GTIN as 14 digits, or None. Accepts GTIN-8, -12, -13 and -14."""
    if not raw:
        return None
    digits = "".join(ch for ch in raw if ch.isdigit())
    if len(digits) not in (8, 12, 13, 14):
        return None
    gtin14 = digits.zfill(14)
    body, check = gtin14[:-1], int(gtin14[-1])
    total = sum(int(d) * (3 if i % 2 == 0 else 1) for i, d in enumerate(body))
    return gtin14 if (10 - total % 10) % 10 == check else None

"036000291452" (a 12-digit code) and "0036000291452" both normalise to 00036000291452, so they match. A single mistyped digit fails the check and returns None, so a corrupt identifier falls through to the next layer instead of forcing a bad match.

Manufacturer part numbers (MPNs) are useful too, but only within the same brand, and they are less standardised than GTINs.

Step 2: extract attributes that can rule a match out

Most listings have no usable identifier, so the title has to do the work. Before comparing titles, pull out the attributes that distinguish otherwise identical products: size, pack count and colour. Strip the brand name so it does not inflate similarity.

import re

UNITS = {"ml": ("ml", 1), "l": ("ml", 1000), "cl": ("ml", 10),
         "g": ("g", 1), "kg": ("g", 1000), "gb": ("gb", 1), "tb": ("gb", 1024)}
SIZE = re.compile(r"(?:(\d+)\s*[x×]\s*)?(\d+(?:\.\d+)?)\s*(ml|cl|l|kg|g|gb|tb)\b", re.I)
COLOURS = {"black", "white", "silver", "grey", "gray", "blue", "red", "green", "gold", "pink"}
STOP = {"the", "and", "with", "for", "by", "new", "pack", "of"}

def extract(title: str, brand: str) -> dict:
    """Pull comparable attributes out of a product title."""
    attrs = {}
    m = SIZE.search(title)
    if m:
        count = int(m.group(1) or 1)
        unit, factor = UNITS[m.group(3).lower()]
        attrs["size"] = (round(float(m.group(2)) * factor * count, 2), unit)
        attrs["pack"] = count
        title = title[:m.start()] + " " + title[m.end():]
    brand_words = set(re.findall(r"[a-z0-9]+", brand.lower()))
    words = {t for t in re.findall(r"[a-z0-9]+", title.lower())
             if t not in STOP and t not in brand_words}
    colour = words & COLOURS
    if colour:
        attrs["colour"] = "grey" if "gray" in colour else sorted(colour)[0]
    attrs["tokens"] = words - COLOURS
    return attrs

"Sparkling Water 6 x 330ml" becomes a size of 1,980 ml, a pack of 6 and the tokens {"sparkling", "water"}. Sizes are converted to a base unit, so 1 l and 1000 ml compare as equal.

Step 3: score, with conflicts first

The scorer applies the layers in order. Valid GTINs decide outright. Then any hard conflict returns zero: a different brand, size, pack, colour, or a differing model word ("pro" against "lite") or model number. Only then does it compute a fuzzy score from token overlap and character similarity.

from difflib import SequenceMatcher

MODEL_WORDS = {"pro", "lite", "max", "mini", "plus", "ultra", "air", "se", "xl"}

def jaccard(a: set, b: set) -> float:
    return len(a & b) / len(a | b) if a | b else 0.0

def match_score(a: dict, b: dict) -> tuple[float, str]:
    """Score two product records from 0 to 1, with the reason."""
    ga, gb = normalise_gtin(a.get("gtin")), normalise_gtin(b.get("gtin"))
    if ga and gb:
        return (1.0, "gtin_equal") if ga == gb else (0.0, "gtin_conflict")

    if a["brand"].strip().lower() != b["brand"].strip().lower():
        return 0.0, "brand_conflict"
    xa, xb = extract(a["title"], a["brand"]), extract(b["title"], b["brand"])
    for attr in ("size", "pack", "colour"):
        if attr in xa and attr in xb and xa[attr] != xb[attr]:
            return 0.0, f"{attr}_conflict"
    differing = xa["tokens"] ^ xb["tokens"]
    if differing & MODEL_WORDS or any(ch.isdigit() for t in differing for ch in t):
        return 0.0, "model_conflict"
    if a.get("mpn") and a.get("mpn") == b.get("mpn"):
        return 0.95, "mpn_equal"

    token_sim = jaccard(xa["tokens"], xb["tokens"])
    char_sim = SequenceMatcher(None, " ".join(sorted(xa["tokens"])),
                               " ".join(sorted(xb["tokens"]))).ratio()
    shared = sum(attr in xa and attr in xb for attr in ("size", "colour"))
    score = 0.6 * token_sim + 0.3 * char_sim + 0.05 * shared
    return round(min(score, 0.9), 3), "fuzzy"

def decide(score: float, auto: float = 0.85, review: float = 0.6) -> str:
    if score >= auto:
        return "match"
    if score >= review:
        return "review"
    return "no_match"

Run against sample listings, it behaves the way a careful person would:

"Acme Wireless Earbuds Pro, Black" against "Wireless Earbuds Pro (Black) by Acme" scores 0.9: match.
The same Pro earbuds in black against white: colour conflict, no match.
The Pro against the Lite: model conflict, no match, despite near-identical titles.
A 6 x 330ml pack against a single 330ml can: size conflict, no match.
"Acme True Wireless Earbuds Pro Black" scores 0.767: review. Probably the same product, but the extra word is worth a human glance.

Fuzzy scores are capped at 0.9, below a GTIN match, so a title can never be more trusted than a verified identifier. In production, add a blocking step first (compare only listings with the same brand and category) so you are not scoring every pair in the catalogue.

Step 4: thresholds and the review queue

The thresholds in decide are a business decision, not a constant. A pricing team acting automatically on matched prices needs very high precision; an analyst exploring a category can live with more review. Send the middle band to a review queue, and feed every human decision back as a labelled pair. Over time, those labels become the test set that tells you whether a change to the matcher made things better or worse.

Step 5: measure precision and recall

You cannot improve a matcher you do not measure. Score a set of hand-labelled pairs and compute precision (how many automatic matches were right) and recall (how many true matches were found automatically):

def evaluate(labelled: list[tuple[dict, dict, bool]], auto: float) -> tuple[float, float]:
    """Precision and recall of automatic matches against hand-labelled pairs."""
    tp = fp = fn = 0
    for a, b, is_match in labelled:
        predicted = match_score(a, b)[0] >= auto
        tp += predicted and is_match
        fp += predicted and not is_match
        fn += (not predicted) and is_match
    precision = tp / (tp + fp) if tp + fp else 1.0
    recall = tp / (tp + fn) if tp + fn else 1.0
    return round(precision, 3), round(recall, 3)

On six labelled sample pairs, an automatic threshold of 0.85 gave precision 1.0 and recall 0.333. Lowering it to 0.65 gave precision 1.0 and recall 1.0. On real catalogues the trade-off is never that clean, which is exactly why you measure it. Track these numbers per category alongside your other data quality metrics, because a matcher tuned on electronics will behave differently on groceries.

Practical notes

• Keep the reason with the score. gtin_conflict and size_conflict are far more useful in a review queue than a bare zero.
• Store matches as data. Keep a match table with both listing IDs, the score, the reason, the decision and who or what made it, so you can audit and reverse decisions.
• Re-match when listings change. Titles and identifiers get edited; a match from six months ago may no longer hold.
• Extend the attribute list per category. Storage capacity for electronics, shade for cosmetics, size and fit for clothing.
• Collect responsibly. Product listings come from public pages; collect at a polite rate and respect each site's terms and robots.txt.
• Wrapping up

Reliable product matching is layered. Trust validated GTINs above everything, rule out matches on hard attribute conflicts before looking at similarity, score what remains, and let confidence thresholds split automatic matches from human review. Then measure precision and recall on labelled pairs, and tune by category. Every price comparison you build afterwards will be only as good as this step.

FAQ

How do you match products across different websites?

Start with validated identifiers such as GTINs. Where they are missing, extract attributes like brand, size, pack count, colour and model, rule out pairs that conflict on any of them, and then score title similarity. Use confidence thresholds to separate automatic matches from pairs that need human review.

What is a GTIN check digit?

The final digit of a GTIN is calculated from the other digits using a weighted sum (alternating weights of 3 and 1, starting from the right of the remaining digits). Recalculating it is a quick way to reject mistyped or corrupted identifiers before using them to match products.

How do you measure product matching accuracy?

Compare automatic matches against a set of hand-labelled pairs and calculate precision, the share of automatic matches that are correct, and recall, the share of true matches found. Tune thresholds per category to balance the two.