DEV Community

Cover image for Marketplace Catalog QC: Image Checks, Duplicates, Attributes
Muhammad Ali Mughal
Muhammad Ali Mughal

Posted on

Marketplace Catalog QC: Image Checks, Duplicates, Attributes

A marketplace catalog decays the moment the second seller uploads a spreadsheet. Within weeks you have the same phone listed nine times, a "red" filter that misses half the red shirts because sellers typed "Maroon", and product photos with watermarks from a competitor's site. Search gets worse, buyers lose trust, and your ops team spends its days approving listings by hand. Catalog quality control is not a moderation team problem. It is a pipeline you design into the platform.

I run Loop Origin, a software studio, and as a Principal Software Engineer I have spent nine years building e-commerce and multi-vendor platforms in Laravel, Vue, Nuxt and PostgreSQL. I wrote this with help from AI tools and checked every query and code sample myself. This guide covers how I structure automated catalog checks: category attribute rules, image checks, duplicate detection, and the review workflow that ties them together.

What is catalog quality control in a multi-vendor marketplace?

Direct answer: Catalog quality control is an automated pipeline that checks every seller listing before it goes live: required attributes per category, normalized values, image size and content checks, and duplicate detection by identifier, title similarity and image hash. Listings that pass publish automatically; the rest go to a review queue with a specific reason code.

The four layers of checks

I split checks into layers that run in order, cheapest first, so expensive work only runs on listings that already passed the basics.

Layer What it checks Cost to run Blocks publish? Typical tool
1. Schema Required attributes per category, types, allowed values Very low Yes Category rules stored as data, validated in PHP
2. Normalization Units, colors, sizes, brand names mapped to canonical values Low No, it fixes or flags Lookup tables plus synonym lists
3. Images Count, dimensions, aspect ratio, file type, blur, near-duplicate images Medium Yes for hard failures Queue worker with an image library and perceptual hashing
4. Duplicates Same GTIN, similar titles in the same category, same image hash Medium Routes to review PostgreSQL unique indexes, pg_trgm, hash comparison

Running them as separate jobs also gives you clear reason codes. "Rejected" tells a seller nothing. "Missing required attribute: screen_size" tells them what to fix.

Layer 1: category attribute rules as data

The worst catalog designs hard-code rules in controllers. Every new category then needs a deploy. I store rules per category in a table so the catalog team can edit them.

CREATE TABLE category_attribute_rules (
 id            bigserial PRIMARY KEY,
 category_id   bigint NOT NULL REFERENCES categories(id),
 attribute     text   NOT NULL,          -- e.g. 'color', 'screen_size_in'
 data_type     text   NOT NULL CHECK (data_type IN ('text','number','enum','boolean')),
 required      boolean NOT NULL DEFAULT false,
 allowed       jsonb,                    -- enum values, or {"min":1,"max":100}
 facet         boolean NOT NULL DEFAULT false,
 UNIQUE (category_id, attribute)
);
Enter fullscreen mode Exit fullscreen mode

The validator is small. It loads rules for the listing's category and returns a list of failures, never throwing on the first one, so the seller sees everything at once.

final class AttributeValidator
{
   public function validate(Listing $listing): array
   {
       $errors = [];
       $rules  = CategoryAttributeRule::where('category_id', $listing->category_id)->get();

       foreach ($rules as $rule) {
           $value = $listing->attributes[$rule->attribute] ?? null;

           if ($rule->required && ($value === null || $value === '')) {
               $errors[] = ['code' => 'ATTR_MISSING', 'attribute' => $rule->attribute];
               continue;
           }
           if ($value === null) continue;

           if ($rule->data_type === 'enum' && !in_array($value, $rule->allowed, true)) {
               $errors[] = ['code' => 'ATTR_NOT_ALLOWED', 'attribute' => $rule->attribute, 'value' => $value];
           }
           if ($rule->data_type === 'number') {
               $min = $rule->allowed['min'] ?? null; $max = $rule->allowed['max'] ?? null;
               if (!is_numeric($value) || ($min !== null && $value < $min) || ($max !== null && $value > $max)) {
                   $errors[] = ['code' => 'ATTR_OUT_OF_RANGE', 'attribute' => $rule->attribute, 'value' => $value];
               }
           }
       }
       return $errors;
   }
}
Enter fullscreen mode Exit fullscreen mode

Mark facet attributes (color, size, brand) as required in categories where buyers filter on them. A filter is only as good as the share of listings that have the value.

Layer 2: normalization before validation fails

Sellers will write "Maroon", "Burgundy", "dark red" and "RED". Rejecting all of them annoys good sellers. Map them instead.

  • Keep a synonym table: attribute, raw_value, canonical_value, scoped by category when meaning differs.
  • Normalize case, whitespace and units before lookup ("15.6 inch", "15.6in", 15.6" all become 15.6).
  • Store both the raw and the canonical value, so you can audit mappings and the seller still sees what they typed.
  • Log every unmapped value. A weekly review of the top unmapped values is the cheapest catalog improvement you can make.

The same applies to brands. "Samsung", "SAMSUNG" and "Samsung Electronics" should resolve to one brand ID, or your brand filter splits into three.

Layer 3: image checks

Images go through a queued job after upload, because they are slow. These are the checks I run, in order:

  1. File type and size: accept JPEG, PNG and WebP; reject files over your limit before decoding.
  2. Dimensions and aspect ratio: a minimum shortest side, and a ratio window per category. Fashion often wants portrait; electronics often square.
  3. Count: a minimum number of images for categories where buyers need angles.
  4. Blur: a variance-of-Laplacian score below a threshold flags soft images for review rather than rejecting them, because thresholds vary by product.
  5. Perceptual hash: compute a pHash and store it as a 64-bit integer. It powers duplicate image detection in layer 4.

Two nearly identical white sneakers side by side on a light grey surface
AI-generated image.

Watermarks and text overlays are harder. You can add an OCR or vision-model check later, but start by flagging them through buyer reports and reviewer reason codes, and measure how common they actually are before paying for a model.

Layer 4: duplicate detection

Duplicates come in three shapes, and each needs a different check.

Duplicate type Signal Check Action
Same product, same seller Same seller, same GTIN or near-identical title Unique index on (seller_id, gtin); title similarity within seller Block, ask seller to edit existing listing
Same product, different sellers Same GTIN or very similar title plus same brand GTIN match across sellers; pg_trgm similarity Attach as a new offer on one product page
Copied listing Same image hash as another seller's listing Hamming distance on pHash Route to review

For title similarity, PostgreSQL's pg_trgm extension is enough for most marketplaces I have built, long before you need a search cluster.

CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX listings_title_trgm ON listings USING gin (normalized_title gin_trgm_ops);

SELECT id, seller_id, normalized_title,
      similarity(normalized_title, :title) AS score
FROM listings
WHERE category_id = :category_id
 AND brand_id    = :brand_id
 AND normalized_title % :title
ORDER BY score DESC
LIMIT 5;
Enter fullscreen mode Exit fullscreen mode

Normalize titles first: lowercase, strip punctuation, remove seller-specific noise like "Free shipping" or "Best price". Tune the similarity threshold on your own data by sampling pairs and labeling them, not by copying a number from a blog post.

For images, compare pHashes by Hamming distance. On a small catalog a direct comparison inside the category is fine; on a large one, bucket hashes by a prefix first so you only compare candidates.

The review workflow

Automation decides most listings. People decide the rest, and the workflow matters as much as the checks.

  • States: draft live, or checking in_review (ops decides).
  • Reason codes: every failure has a code and a seller-facing message. Reviewers pick codes, not free text.
  • Re-check on edit: any edit to attributes, title or images sends the listing back through the pipeline.
  • Seller quality score: track the share of a seller's listings that pass first time. Trusted sellers can skip manual review for low-risk categories.
  • Audit log: who approved what and why. You will need it when a brand complains about counterfeits.

Build vs. buy for catalog QC

Option Fits when Watch out for
Rules in your own Laravel/PostgreSQL stack Up to a large catalog with clear categories; you want full control You own threshold tuning and synonym lists
Marketplace platform built-in moderation You are on a platform that already has it and your rules are simple Limited custom rules and reason codes
Product information management (PIM) tool Many categories, a catalog team, and feeds to several channels License cost and another system to sync
Vision or language model checks Watermarks, prohibited items, category mismatch at scale Per-call cost, false positives, need for human review

My default for a new marketplace is the first row, with a clear place to plug in model-based checks later. I covered the payment side of the same platforms in Multi-Vendor Marketplace Payments: Ledgers & Payouts and the cart side in Multi-Vendor Checkout: Splitting One Cart by Seller.

Failure modes I see

  • Rules that only block: a strict validator with no normalization pushes good sellers away. Normalize first, reject last.
  • Bulk upload bypass: the web form validates, the CSV importer does not. Route both through the same pipeline.
  • Silent threshold drift: someone lowers the similarity threshold to clear a queue and duplicates flood in. Store thresholds as config with an audit trail.
  • Duplicates merged too eagerly: two products with the same title but different storage sizes get merged. Require attribute agreement, not just title similarity, before merging.
  • No feedback loop: reviewers keep fixing the same unmapped values by hand. Feed their decisions back into the synonym table.

Rollout checklist

  1. Define required and facet attributes for your top categories first.
  2. Build the synonym table and log unmapped values from day one.
  3. Run every check in shadow mode on existing listings and review the results before enforcing.
  4. Turn on hard blocks for schema and image failures; route duplicates to review.
  5. Give every failure a reason code and a seller-facing message.
  6. Send bulk imports through the same pipeline as the web form.
  7. Review unmapped values, top reason codes and review-queue age weekly.
  8. Add a seller quality score and trusted-seller fast lane once you have data.

FAQ

What is the first catalog check to build in a new marketplace?

Category attribute rules. Required attributes and allowed values for your top categories improve filters and search immediately, and they are cheap to run on every listing before any image or duplicate check.

How do you detect duplicate products from different sellers?

Match on GTIN or barcode where sellers provide it, then on normalized title similarity within the same category and brand, then on image hash. When products truly match, attach the second seller as a new offer on the same product page rather than a new listing.

Is PostgreSQL enough for duplicate title detection?

For most marketplaces, yes. The pg_trgm extension with a GIN index finds similar titles quickly when you filter by category and brand first. Move to a dedicated search engine when you also need relevance ranking and typo-tolerant search at large scale.

Should listings with low-quality images be rejected automatically?

Reject hard failures like wrong file type or tiny dimensions. Flag soft issues like blur or possible watermarks for review, because automatic thresholds produce false positives that frustrate good sellers.

How do you stop CSV bulk uploads from skipping checks?

Make the importer create listings in the same checking state and dispatch the same jobs as the web form. Never write directly to live from an import, and report per-row reason codes back to the seller.

When is a PIM tool worth it for a marketplace?

When you have many categories, a dedicated catalog team and feeds to several sales channels. For a single marketplace with clear categories, rules inside your own stack are usually simpler and cheaper to run.

The short version

Treat catalog quality as a pipeline: rules as data, normalization before rejection, queued image checks, duplicate detection with plain PostgreSQL, and a review workflow with reason codes. Run it in shadow mode first, then enforce.


Follow & Connect

Top comments (0)