This is Part 1 of a two-part series on building an automated product image-description mismatch detector for online marketplaces. Part 1 covers the problem, the CLIP baseline, and BLIP. Part 2 covers the OCR + Visual Question Answering pipeline that followed.
The Problem No One Notices Until It's Too Late
Imagine shopping online for a specific brand of milk. The image shows a Prairie Farms carton, but the description says Organic Valley. Or the image matches, but the size is wrong. You hesitate, second-guess yourself, and maybe abandon the cart entirely.
Prairie Farms Whole Milk — 1 quart carton used throughout this article
This small moment of doubt is not an edge case — it is one of the most common pain points in e-commerce. When product images and text descriptions are out of sync, the cost is real: buyers lose confidence, return rates climb, and platforms take a reputational hit.
Across major marketplaces, mismatches between images and descriptions rank among the leading causes of customer frustration, alongside missing or unclear product information [1]. On Amazon, a significant share of returns and complaints stems from products that look different from their listing [2]. On eBay, buyers frequently report disputes triggered by images that don't match what arrived [1]. On Instacart, grocery listings sometimes carry outdated or incorrect images, leading customers to select the wrong item — particularly problematic when products look visually similar [3].
The core issue is scale. Large marketplaces host millions of listings, many uploaded by third-party sellers with inconsistent quality control. Manual review at that scale is not feasible. What is needed is an automated system that can reliably flag mismatched listings before customers encounter them.
This is the problem I set out to tackle in this project.
Why This Is Harder Than It Looks
At first glance, matching an image to a description seems straightforward. But real-world product data is deceptively tricky:
- Brand mismatches: Two cartons of milk may look nearly identical in shape and color, but belong to different brands. The distinction lives entirely in the label text — a few pixels in a small font.
- Size mismatches: "8 oz" vs. "16 oz" may show visually identical packaging, differing only in fine print on the back of the label.
- Variant mismatches: "Salted butter" vs. "unsalted butter" — same product, different SKU, meaningfully different for the customer.
- Obvious mismatches: An image of a chicken breast paired with a butter description — easy for a human, but a system still needs to catch it reliably at scale.
Traditional image classification or keyword matching falls apart on the first three categories. What is needed is a model that understands both images and text in a unified way.
Phase 1: CLIP as a Zero-Shot Baseline
CLIP (Contrastive Language–Image Pretraining), developed by OpenAI, was the natural starting point. CLIP is trained on hundreds of millions of image-text pairs to learn a shared embedding space — a mathematical representation where semantically related images and text end up close together, and unrelated ones end up far apart.
In practice:
- An image of a milk carton and the text "whole milk" will have a high cosine similarity score.
- The same image paired with "a cat playing with a ball" will have a very low score.
- Subtle mismatches — wrong brand, wrong size — fall somewhere in between.
How it is implemented
from transformers import CLIPProcessor, CLIPModel
from src.config import Config
from src.utils import fetch_image_from_url
def compute_clip_similarity(configs: Config, image_url: str, description: str) -> float:
device = "mps" if torch.backends.mps.is_available() else "cpu"
model = CLIPModel.from_pretrained(configs.clip_model).to(device)
processor = CLIPProcessor.from_pretrained(configs.clip_processor)
img = fetch_image_from_url(image_url)
inputs = processor(text=[description], images=img, return_tensors="pt", padding=True).to(device)
with torch.no_grad():
outputs = model(**inputs)
image_emb = outputs.image_embeds / outputs.image_embeds.norm(dim=-1, keepdim=True)
text_emb = outputs.text_embeds / outputs.text_embeds.norm(dim=-1, keepdim=True)
return (image_emb @ text_emb.T).item()
The similarity score is mapped to a label:
| Score | Label |
|---|---|
| > 0.30 | Likely match |
| 0.20 – 0.30 | Uncertain |
| < 0.20 | Likely mismatch |
What CLIP gets right — and where it falls short
Testing against a Prairie Farms whole milk image with several descriptions:
| Description | Score | Label |
|---|---|---|
| One quart of whole milk from Prairie Farms (correct) | 0.31 | Likely match |
| One quart of whole milk from Organic Valley (wrong brand) | 0.29 | Uncertain |
| Half gallon of whole milk from Prairie Farms (wrong size) | 0.30 | Uncertain |
| A cat playing with a ball (obvious mismatch) | 0.11 | Likely mismatch |
CLIP catches the obvious mismatch confidently. But wrong brand and wrong size both land in "Uncertain" — CLIP recognises the product type correctly but misses the brand and quantity distinction. This is a known limitation: CLIP encodes descriptions holistically, so quantity words like "half gallon" or brand names like "Organic Valley" may not shift the embedding enough to trigger a clear mismatch signal.
CLIP is a useful zero-shot baseline for catching gross mismatches, but fine-grained product-level distinctions remain a challenge with similarity scoring alone.
Phase 2: BLIP for Richer Image Understanding
The natural next step was BLIP (Bootstrapping Language-Image Pre-training), which brought two new capabilities:
2a. Image Captioning with BLIP-2
Instead of comparing embeddings directly, a captioning model can describe what it sees in an image. This makes the model's perception explicit and debuggable.
The original blip-image-captioning-base had a significant failure mode on text-heavy product images — it produced repetitive output like "classic classic classic classic..." when encountering product labels with bold text. The fix was to upgrade to Salesforce/blip2-opt-2.7b, which uses a large language model (OPT-2.7B) as its language backbone and generates coherent captions.
from transformers import Blip2Processor, Blip2ForConditionalGeneration
def generate_blip_caption(configs: Config, image_url: str) -> str:
device = "mps" if torch.backends.mps.is_available() else "cpu"
processor = Blip2Processor.from_pretrained(configs.blip2_model)
model = Blip2ForConditionalGeneration.from_pretrained(
configs.blip2_model, torch_dtype=torch.float16
).to(device)
img = fetch_image_from_url(image_url)
inputs = processor(images=img, return_tensors="pt").to(device, torch.float16)
with torch.no_grad():
output = model.generate(**inputs, max_new_tokens=50)
return processor.decode(output[0], skip_special_tokens=True)
Results on three products:
| Product | Actual description | BLIP-2 caption |
|---|---|---|
| Prairie Farms Whole Milk | one quart of whole milk from Prairie Farms | a carton of milk with the label on it |
| Amazon Fresh Chicken Breast | boneless skinless chicken breast, 30 oz | a package of chicken breast |
| Land O Lakes Salted Butter | salted butter, 8 oz | land o lakes salted butter |
Left to right: Prairie Farms Whole Milk, Amazon Fresh Chicken Breast, Land O Lakes Salted Butter — the three products used in all experiments
BLIP-2 correctly reads "Land O Lakes salted butter" from the label — noticeably better than the base model. But it loses granular detail like size and variant for the milk and chicken.
2b. Image-Text Matching (ITM)
BLIP's ITM head is more directly useful for mismatch detection. Rather than generating text, it outputs a probability score between 0 and 1 representing how well an image and text match — directly interpretable without calibration.
from transformers import BlipProcessor, BlipForImageTextRetrieval
def compute_blip_itm_score(configs: Config, image_url: str, description: str) -> float:
device = "mps" if torch.backends.mps.is_available() else "cpu"
processor = BlipProcessor.from_pretrained(configs.blip_itm_model)
model = BlipForImageTextRetrieval.from_pretrained(configs.blip_itm_model).to(device)
img = fetch_image_from_url(image_url)
inputs = processor(images=img, text=description, return_tensors="pt").to(device)
with torch.no_grad():
outputs = model(**inputs, use_itm_head=True)
score = torch.nn.functional.softmax(outputs.itm_score, dim=1)[:, 1].item()
return score
| Description | ITM Score | Label |
|---|---|---|
| Correct (Prairie Farms milk) | 0.9998 | Likely match |
| Wrong brand (Organic Valley) | 0.7872 | Likely match |
| Wrong size (two quarts) | 0.9996 | Likely match |
| Obvious mismatch (cat playing) | 0.0000 | Likely mismatch |
ITM is excellent at catching obvious mismatches. However, wrong brand and wrong size both score very high — the model is too lenient for fine-grained product matching. A wrong brand gets a 0.79 ITM score, which would pass as a likely match in a marketplace context.
This revealed a fundamental limitation: both CLIP and BLIP treat the description as a whole. Neither one systematically checks individual product attributes like brand, size, and variant in isolation.
The Lesson from Phase 1 and 2
CLIP and BLIP are powerful general-purpose models, but neither was designed to answer the specific question a marketplace needs: "Does the brand in the image match the brand in the description? Does the size match?"
What is needed is not a similarity score — it is a structured, per-feature comparison. That insight drives the approach in Part 2.
Why It Matters
Product content quality is not a niche concern. Research consistently links inconsistencies between images and text to reduced customer confidence and lower purchase intent [4]. Platforms that invest in automated content quality tooling see measurable improvements in conversion rates, return rates, and seller accountability.
An automated mismatch detection pipeline does not replace human review — it scales it. By surfacing the listings most likely to be wrong, it directs human attention to where it matters most, at a fraction of the cost of manual auditing.
What's Next
In Part 2, I replace free-form similarity scoring with a feature dictionary approach:
- OCR (Optical Character Recognition) reads brand names, sizes, and variants directly from the product label as text
- VQA (Visual Question Answering) answers targeted questions about visual features the OCR cannot read (container type, product category) and acts as a fallback when OCR misses stylised fonts or logos
- A hard veto on all features ensures any single wrong attribute flags the listing — because in a marketplace, a wrong size is just as damaging as a wrong product
→ Continue to Part 2: OCR + VQA
Code and Experiments
All code and Jupyter notebooks for this project are available on GitHub:
github.com/ebiarian/product-image-description-alignment
References
- Buzzin Content. 91.3% of E-Commerce Brands Failed to Address Real Customer Pain Points. https://www.buzzincontent.com/story/91-3-e-commerce-brands-failed-to-address-real-customer-pain-points-in-pdp-content-odn-report/
- Besedo. The Effects of Fraud and Poor Content Quality on Online Marketplaces. https://besedo.com/blog/report-the-effects-of-fraud-and-poor-content-quality-on-online-marketplaces/
- Perpetua Help. Missing or Incorrect Product Images. https://help.perpetua.io/en/articles/4626446-missing-or-incorrect-product-images
- ScienceDirect. Inconsistencies between images and text in online shopping. https://www.sciencedirect.com/science/article/abs/pii/S0969698917307142
Top comments (0)