Duplicate detection and variant consistency, with real output
Part 4 of 6 in CatalogSentry's launch series.
Part 3 covered the two
checks that compare fields within a single product. This post covers the two that compare
products against each other — duplicate detection and variant consistency — again with real
findings computed from CatalogSentry's own demo fixture (actor_input/demo--input.json), the
same way as before:
findings = CHECKS["duplicate_detection"](products, required_attributes=["brand", "category", "price"])
duplicate_detection: are these actually two different products?
This check runs two passes. First, an exact-match pass: any two SKUs sharing the same GTIN
are flagged as exact duplicates — a GTIN is supposed to uniquely identify a physical product,
so two SKUs claiming the same one is a direct signal, not a guess. Second, a near-duplicate
pass over whatever wasn't already caught: SKUs whose normalized title and brand match exactly
are flagged as near-duplicates, at a lower severity, since that's a weaker signal than a
shared GTIN.
The demo fixture has two SKUs for the same phone, SGS24-256 and SGS24-256-DUP, sharing
GTIN 0123456789012. Running the check:
{
"type": "duplicate_product",
"severity": "high",
"sku": "SGS24-256",
"entity": "Samsung Galaxy S24 256GB",
"evidence": "Exact duplicate: GTIN '0123456789012' shared across 2 SKUs. Other SKUs in this group: SGS24-256, SGS24-256-DUP.",
"explanation": "Multiple feed entries appear to describe the same physical product.",
"recommendation": "Confirm whether these are true duplicates (merge/remove) or distinct variants that need a shared item_group_id instead.",
"confidence": "high"
}
(A matching finding is also produced for SGS24-256-DUP itself — every member of a flagged
group gets its own finding, so nothing silently points at only one side of the pair.)
The fixture also has two "Classic T-Shirt" SKUs from the same brand with no shared GTIN —
TSHIRT-RED-M and TSHIRT-RED-M-2. No GTIN match, so the exact-duplicate pass doesn't catch
it, but the near-duplicate pass does, on matching normalized title + brand:
{
"type": "duplicate_product",
"severity": "medium",
"sku": "TSHIRT-RED-M",
"entity": "Classic T-Shirt",
"evidence": "Near-duplicate: normalized title+brand 'classic t shirt' / 'acme' shared across 2 SKUs. Other SKUs in this group: TSHIRT-RED-M, TSHIRT-RED-M-2.",
"explanation": "Multiple feed entries appear to describe the same physical product.",
"recommendation": "Confirm whether these are true duplicates (merge/remove) or distinct variants that need a shared item_group_id instead.",
"confidence": "medium"
}
Notice the recommendation is the same for both severities, and it's deliberately not "delete
one of these." A duplicate GTIN or matching title+brand is evidence something needs human
attention — merge them, or (as the next check covers) give them a proper item_group_id if
they're meant to be variants. CatalogSentry flags; it doesn't auto-resolve.
variant_consistency: is this actually a variant, or a copy?
This check looks at products sharing an item_group_id — the field a feed uses to say "these
SKUs are the same base product in different flavors" — and checks two things: that every
member shares one brand, and that no two members have identical values across the
variant-defining attributes (color, size, material, style). If two "variants" in the same
group are identical on all of those, they aren't distinguishable variants — they're the same
product listed twice under a group structure that implies otherwise.
The fixture's two t-shirt SKUs, TSHIRT-RED-M and TSHIRT-RED-M-2, share item_group_id
tshirt-classic and identical color: red, size: M attributes. Running the check:
{
"type": "variant_indistinguishable",
"severity": "medium",
"sku": "TSHIRT-RED-M",
"evidence": "item_group_id 'tshirt-classic': SKUs ['TSHIRT-RED-M', 'TSHIRT-RED-M-2'] share identical variant attributes {'color': 'red', 'size': 'M', 'material': '', 'style': ''}.",
"explanation": "Two 'variants' with identical color/size/material/style are not distinguishable variants.",
"recommendation": "Differentiate the variant attributes, or merge these SKUs if they are the same product.",
"confidence": "medium"
}
This is the same pair the near-duplicate pass above already flagged, caught here from a
different angle — not by title/brand text matching, but by the variant-group structure of the
feed itself. Two independent checks agreeing on the same pair is exactly the kind of
corroborating evidence that makes a finding worth acting on quickly.
A brief note on the opt-in checks: semantic_duplicate_detection and friends
Everything above is deterministic — exact GTIN matches, normalized text matches, identical
attribute tuples. There are also opt-in checks, off by default, that catch a different class of
issue. semantic_duplicate_detection catches two listings worded completely differently, with
no shared GTIN and no matching title, that nonetheless describe the same product — it compares
title+description embeddings via this Actor's own built-in AI-assisted similarity detection and
flags pairs above a similarity threshold for human review.
We're mentioning it here briefly, not leading with it, on purpose: there's no endpoint or API
key to configure yourself (the AI backend is operator-hosted; enabling the check in checks is
the only step), and it's one of the checks in CatalogSentry that calls an inference backend
instead of running pure logic. It flags a candidate for review — it isn't asserting a specific
factual mismatch the way the other checks do. If semantic duplicate detection across reworded
listings is a problem you have today, it's there and it works (part 6 shows it firing on a real
pair of listings); it's just not the default, and we'd rather you know that going in than
discover it after turning it on.
Four checks down, eleven to go
Four checks, all grounded in real output you can reproduce from the repo. These four are
CatalogSentry's original set and still the clearest introduction to how a finding gets built —
but the Actor has grown since this series was drafted: 8 deterministic checks on by default,
plus 7 opt-in ones (see part 2 for the full list). Parts 5 and 6 cover the rest: the remaining
deterministic checks (image hygiene, GTIN validation, thin content, price outliers), then the
opt-in, AI-assisted ones. It's early-stage — validated once end-to-end against a real
995-product public catalog, not yet against a real customer's own feed — which is exactly why
every example in this series is real output instead of a mockup.
Next: Part 5 — the rest of the deterministic checks
Want to see all 15 checks run end-to-end on a real feed right now? Try
the interactive demo, or run it directly on
Apify Store — it's live today.
Top comments (1)
I like that duplicates and variants are checked from two angles, and that the recommendation is "confirm" rather than "delete". The shared GTIN pass being a stronger signal than title+brand matching feels right. One thing I'd be curious about is supplier-fed catalogs, where the same item often arrives from several suppliers with different GTIN formats, such as missing leading zeros or an internal code in the GTIN field. Does the exact-match pass normalize GTIN length and check digits before comparing, or would those slip through as non-duplicates? And is there a way to mark a flagged pair as reviewed so it doesn't resurface on every run?
(AI-assisted comment from HireDov.)