DEV Community: Ebrahim Arian

When the Picture Doesn't Match the Label — Part 2: Feature-Based Detection with OCR and VQA

Ebrahim Arian — Thu, 23 Jul 2026 02:43:34 +0000

This is Part 2 of a two-part series. Part 1 covered the problem, the CLIP baseline, and BLIP. This part covers the OCR + VQA pipeline that replaced them for fine-grained product feature matching.

The Problem with Scores

At the end of Part 1, the conclusion was clear: CLIP and BLIP are good at catching obvious mismatches but fail at the fine-grained ones that matter most in a marketplace. A wrong brand scores 0.79 on BLIP ITM — comfortably inside "Likely match" territory. A wrong size is nearly invisible to both models.

The reason is architectural. Both CLIP and BLIP compare an image and a description as whole units. They never ask: does the brand in the image match the brand in the description? They encode everything together and produce a single score. That score cannot tell you which attribute caused a mismatch.

What a marketplace actually needs is not a similarity score. It needs answers to specific questions:

Is this the right brand?
Is this the right size?
Is this the right variant?

That insight led to a completely different approach.

The Three Products Used in This Article

All experiments in this article are run against three real grocery products — chosen to represent different label types, packaging materials, and mismatch challenges:

Prairie Farms Whole Milk (1 quart carton) — a text-heavy label with a stylised brand logo, nutritional facts on the back, and a clear front panel. Useful for testing brand and variant detection.
Amazon Fresh Boneless Skinless Chicken Breast (30 oz plastic container) — minimal label text, brand printed on a sticker. Tests whether the model can identify both the brand and the specific cut.
Land O Lakes Salted Butter (8 oz paper wrapper) — compact packaging with multiple size representations printed on the label (8 oz, 226.8 g, 4 half sticks). Tests size matching edge cases.

The Feature Dictionary Approach

Instead of scoring image against description holistically, the idea is to define a feature dictionary — a structured set of attributes that any product listing should have — and verify each one independently.

description_features = {
    "product_type": "milk",
    "brand":        "prairie farms",
    "size":         ["1 quart", "946ml", "1 qt"],  # multiple equivalent representations
    "variant":      "whole",
    "container":    "carton",
}

For each feature, the system extracts the corresponding value from the product image and compares it against the description. If any single feature fails — the listing is flagged.

Two tools do the extraction, each handling what the other cannot:

Tool 1: OCR for Text Features

OCR (Optical Character Recognition) reads printed text directly from an image pixel by pixel and converts it into a string. For product labels, the most important information — brand name, weight, variant — is printed text. Reading it directly is faster and more accurate than asking a vision model to interpret it.

We use EasyOCR (pip install easyocr), a Python library that runs locally with no API calls required.

import easyocr
import numpy as np
from src.utils import fetch_image_from_url

def extract_ocr_text(image_url: str, min_confidence: float = 0.4) -> str:
    img = fetch_image_from_url(image_url)
    img_array = np.array(img)

    reader = easyocr.Reader(["en"], gpu=False, verbose=False)
    results = reader.readtext(img_array)

    texts = [text for _, text, conf in results if conf >= min_confidence]
    return " ".join(texts).lower()

OCR on the Land O Lakes butter label returns:

jarmer-owned landolakes since 1921 tbsp 4 half sticks butter half net wt 8oz (226.8 g) salted

"land o lakes", "butter", "8oz", and "salted" are all there.

The space stripping trick

OCR commonly reads "8oz" as a single token while the description says "8 oz" with a space. A naive substring match would fail. The fix is to strip all spaces before comparing:

ocr_found = any(
    d.replace(" ", "") in ocr_text.replace(" ", "")
    for d in desc_norms
)

Multiple equivalent representations

Size values are inconsistent across listings. A butter listing might say "8 oz", "226.8 g", or "4 half sticks" — all referring to the same product. Requiring an exact match would cause false negatives. The solution is to accept a list of equivalent representations for any feature:

"size": ["8 oz", "226.8 g", "4 half sticks"]

If OCR finds any one of them, the feature passes.

Tool 2: Moondream2 VQA as Fallback

OCR has two hard limits. First, it cannot read stylised logos and custom brand fonts — "Prairie Farms" rendered as a decorative script may return garbled characters or nothing at all. Second, some features are not printed as plain text at all — "container type" is not written on the label; it is a visual property the model must infer.

For these cases, the fallback is VQA (Visual Question Answering) using Moondream2 (1.86B parameters, vikhyatk/moondream2).

VQA models take an image and a natural language question and return a short answer. Unlike captioning — which describes everything and may miss the specific detail you care about — VQA is targeted and controllable.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "vikhyatk/moondream2",
    revision="2025-01-09",
    trust_remote_code=True,
    torch_dtype=torch.float16,
).to(device)
tokenizer = AutoTokenizer.from_pretrained("vikhyatk/moondream2", revision="2025-01-09")

enc_image = model.encode_image(image)   # encode once, ask many questions
answer = model.answer_question(enc_image, "What brand name is shown on this product?", tokenizer)

Why Moondream2?

Model	Size	Why chosen
`blip-vqa-base`	~400MB	Weak instruction-following, poor on product labels
`Salesforce/blip2-opt-2.7b`	~5.5GB	No VQA head, generative only
`vikhyatk/moondream2`	~4GB	Strong instruction-following, efficient on M2, handles logos well

Moondream2 runs in float16 on Apple M2 MPS and uses approximately 4 GB of unified memory — comfortable on a 16 GB machine.

Constrained prompts

Open-ended questions invite unpredictable answers. Constrained prompts force the model to choose from a known vocabulary, which reduces hallucination:

FEATURE_QUESTIONS = {
    "product_type": "Is this milk, coffee, butter, chicken, meat, bread, juice, a beverage, snack, or cleaning product?",
    "container":    "Is this product in a bottle, can, carton, container, canister, bag, or box?",
    "brand":        "What brand name is shown on this product?",
    "size":         "What is the size or quantity? Find a number with these words: oz, quart, gram.",
    "variant":      "What variant or flavor is this? For example: whole, 2%, dark roast, classic roast.",
}

Putting It Together: OCR First, VQA Fallback

The routing logic is simple:

For text features (brand, size, variant, product_type):
  1. Try OCR → if found, done
  2. If not found → ask Moondream2

For visual features (container):
  → Ask Moondream2 directly (not printed as text)

def compare_features(image_features: dict, description_features: dict) -> dict:
    ocr_text = image_features.get("_ocr_text", "")

    for feature, desc_val in description_features.items():
        desc_norms = [v.strip().lower() for v in desc_val] if isinstance(desc_val, list) else [desc_val.lower()]

        if feature in TEXT_FEATURES:
            ocr_found = any(d.replace(" ", "") in ocr_text.replace(" ", "") for d in desc_norms)
            if ocr_found:
                matched, source, img_answer = True, "OCR", "found in OCR"
            else:
                vqa_val = image_features.get(f"_vqa_{feature}", "").lower()
                matched = any(vqa_val in d or d in vqa_val for d in desc_norms)
                source, img_answer = "VQA fallback", vqa_val
        else:
            vqa_val = image_features.get(f"_vqa_{feature}", "").lower()
            matched = any(vqa_val in d or d in vqa_val for d in desc_norms)
            source, img_answer = "VQA", vqa_val

Results on Three Products

Product	brand	product_type	size	variant	container	Overall
Prairie Farms Whole Milk	✓ VQA	✓ OCR	✓ OCR	✓ OCR	✓ VQA	✅ Match
Amazon Fresh Chicken Breast	✓ OCR	✓ OCR	✓ OCR	✓ OCR	✓ VQA	✅ Match
Land O Lakes Salted Butter	✓ OCR	✓ OCR	✓ OCR	✓ OCR	✓ VQA	✅ Match

All correct descriptions matched. Wrong brand, wrong size, and wrong variant correctly triggered mismatches across all three products.

Every Feature Is Critical

An early version of the labelling logic used a ratio: if 75% of features match, the listing passes. This is wrong for a marketplace.

A customer ordering 8 oz of butter who receives 16 oz has received the wrong product. A boneless chicken breast described as bone-in is a real fulfilment failure. There is no threshold at which a wrong size becomes acceptable.

The final labelling logic is a hard veto:

CRITICAL_FEATURES = {"brand", "product_type", "size", "variant", "container"}

def overall_label(feature_results: dict) -> str:
    for feature in CRITICAL_FEATURES:
        if feature in feature_results and not feature_results[feature]["match"]:
            return "Likely mismatch"   # immediate, no exceptions
    return "Likely match"

Any single feature failure returns "Likely mismatch" regardless of what else matched.

Limitations

This approach is meaningfully better than CLIP or BLIP for fine-grained product matching, but several real-world challenges remain:

Image angle: OCR reads only what is visible. If the listing image shows the nutrition facts panel rather than the front label, brand and variant text will be missed entirely. The fix — accepted multiple image URLs per product and merge OCR results — is left for future work.

Image quality: Blurry or compressed images reduce OCR accuracy and may cause Moondream2 to hallucinate answers for features it cannot clearly see.

Prompt coverage: The current VQA prompts cover a handful of product categories. In a real marketplace with thousands of categories, prompts would need to be carefully engineered per category — broader option lists, context-aware phrasing, potentially chain-of-thought reasoning. The current prompts are a proof of concept, not production-ready.

Semantic matching: The comparison logic uses substring matching after space normalisation. "1 qt" and "one quart" are not matched even though they are equivalent. Embedding-based semantic matching would handle this but adds complexity.

Takeaways

The key insight from this project is that the right framing matters more than the right model. CLIP and BLIP are powerful, but they answer the wrong question — "how similar are these overall?" — when the marketplace needs answers to specific questions: wrong brand? wrong size? wrong variant?

Switching to a feature dictionary approach with OCR and VQA fallback made the system interpretable (you can see exactly which feature failed), debuggable (you can inspect the OCR text and VQA answers directly), and strict (a single wrong feature is enough to flag a listing).

The combination of EasyOCR for printed text and Moondream2 for visual reasoning is not perfect — but it is a meaningful step toward a system that could be useful in production.

Code and Experiments

All code, notebooks, and experiment results are available on GitHub:
github.com/ebiarian/product-image-description-alignment

← Back to Part 1: CLIP & BLIP

When the Picture Doesn't Match the Label — Part 1: Using CLIP and BLIP to Catch Product Mismatches

Ebrahim Arian — Thu, 23 Jul 2026 02:40:44 +0000

This is Part 1 of a two-part series on building an automated product image-description mismatch detector for online marketplaces. Part 1 covers the problem, the CLIP baseline, and BLIP. Part 2 covers the OCR + Visual Question Answering pipeline that followed.

The Problem No One Notices Until It's Too Late

Imagine shopping online for a specific brand of milk. The image shows a Prairie Farms carton, but the description says Organic Valley. Or the image matches, but the size is wrong. You hesitate, second-guess yourself, and maybe abandon the cart entirely.

Prairie Farms Whole Milk — 1 quart carton used throughout this article

This small moment of doubt is not an edge case — it is one of the most common pain points in e-commerce. When product images and text descriptions are out of sync, the cost is real: buyers lose confidence, return rates climb, and platforms take a reputational hit.

Across major marketplaces, mismatches between images and descriptions rank among the leading causes of customer frustration, alongside missing or unclear product information [1]. On Amazon, a significant share of returns and complaints stems from products that look different from their listing [2]. On eBay, buyers frequently report disputes triggered by images that don't match what arrived [1]. On Instacart, grocery listings sometimes carry outdated or incorrect images, leading customers to select the wrong item — particularly problematic when products look visually similar [3].

The core issue is scale. Large marketplaces host millions of listings, many uploaded by third-party sellers with inconsistent quality control. Manual review at that scale is not feasible. What is needed is an automated system that can reliably flag mismatched listings before customers encounter them.

This is the problem I set out to tackle in this project.

Why This Is Harder Than It Looks

At first glance, matching an image to a description seems straightforward. But real-world product data is deceptively tricky:

Brand mismatches: Two cartons of milk may look nearly identical in shape and color, but belong to different brands. The distinction lives entirely in the label text — a few pixels in a small font.
Size mismatches: "8 oz" vs. "16 oz" may show visually identical packaging, differing only in fine print on the back of the label.
Variant mismatches: "Salted butter" vs. "unsalted butter" — same product, different SKU, meaningfully different for the customer.
Obvious mismatches: An image of a chicken breast paired with a butter description — easy for a human, but a system still needs to catch it reliably at scale.

Traditional image classification or keyword matching falls apart on the first three categories. What is needed is a model that understands both images and text in a unified way.

Phase 1: CLIP as a Zero-Shot Baseline

CLIP (Contrastive Language–Image Pretraining), developed by OpenAI, was the natural starting point. CLIP is trained on hundreds of millions of image-text pairs to learn a shared embedding space — a mathematical representation where semantically related images and text end up close together, and unrelated ones end up far apart.

In practice:

An image of a milk carton and the text "whole milk" will have a high cosine similarity score.
The same image paired with "a cat playing with a ball" will have a very low score.
Subtle mismatches — wrong brand, wrong size — fall somewhere in between.

How it is implemented

from transformers import CLIPProcessor, CLIPModel
from src.config import Config
from src.utils import fetch_image_from_url

def compute_clip_similarity(configs: Config, image_url: str, description: str) -> float:
    device = "mps" if torch.backends.mps.is_available() else "cpu"
    model = CLIPModel.from_pretrained(configs.clip_model).to(device)
    processor = CLIPProcessor.from_pretrained(configs.clip_processor)

    img = fetch_image_from_url(image_url)
    inputs = processor(text=[description], images=img, return_tensors="pt", padding=True).to(device)

    with torch.no_grad():
        outputs = model(**inputs)
        image_emb = outputs.image_embeds / outputs.image_embeds.norm(dim=-1, keepdim=True)
        text_emb  = outputs.text_embeds  / outputs.text_embeds.norm(dim=-1, keepdim=True)

    return (image_emb @ text_emb.T).item()

The similarity score is mapped to a label:

Score	Label
> 0.30	Likely match
0.20 – 0.30	Uncertain
< 0.20	Likely mismatch

What CLIP gets right — and where it falls short

Testing against a Prairie Farms whole milk image with several descriptions:

Description	Score	Label
One quart of whole milk from Prairie Farms (correct)	0.31	Likely match
One quart of whole milk from Organic Valley (wrong brand)	0.29	Uncertain
Half gallon of whole milk from Prairie Farms (wrong size)	0.30	Uncertain
A cat playing with a ball (obvious mismatch)	0.11	Likely mismatch

CLIP catches the obvious mismatch confidently. But wrong brand and wrong size both land in "Uncertain" — CLIP recognises the product type correctly but misses the brand and quantity distinction. This is a known limitation: CLIP encodes descriptions holistically, so quantity words like "half gallon" or brand names like "Organic Valley" may not shift the embedding enough to trigger a clear mismatch signal.

CLIP is a useful zero-shot baseline for catching gross mismatches, but fine-grained product-level distinctions remain a challenge with similarity scoring alone.

Phase 2: BLIP for Richer Image Understanding

The natural next step was BLIP (Bootstrapping Language-Image Pre-training), which brought two new capabilities:

2a. Image Captioning with BLIP-2

Instead of comparing embeddings directly, a captioning model can describe what it sees in an image. This makes the model's perception explicit and debuggable.

The original blip-image-captioning-base had a significant failure mode on text-heavy product images — it produced repetitive output like "classic classic classic classic..." when encountering product labels with bold text. The fix was to upgrade to Salesforce/blip2-opt-2.7b, which uses a large language model (OPT-2.7B) as its language backbone and generates coherent captions.

from transformers import Blip2Processor, Blip2ForConditionalGeneration

def generate_blip_caption(configs: Config, image_url: str) -> str:
    device = "mps" if torch.backends.mps.is_available() else "cpu"
    processor = Blip2Processor.from_pretrained(configs.blip2_model)
    model = Blip2ForConditionalGeneration.from_pretrained(
        configs.blip2_model, torch_dtype=torch.float16
    ).to(device)

    img = fetch_image_from_url(image_url)
    inputs = processor(images=img, return_tensors="pt").to(device, torch.float16)

    with torch.no_grad():
        output = model.generate(**inputs, max_new_tokens=50)

    return processor.decode(output[0], skip_special_tokens=True)

Results on three products:

Product	Actual description	BLIP-2 caption
Prairie Farms Whole Milk	one quart of whole milk from Prairie Farms	a carton of milk with the label on it
Amazon Fresh Chicken Breast	boneless skinless chicken breast, 30 oz	a package of chicken breast
Land O Lakes Salted Butter	salted butter, 8 oz	land o lakes salted butter

Left to right: Prairie Farms Whole Milk, Amazon Fresh Chicken Breast, Land O Lakes Salted Butter — the three products used in all experiments

BLIP-2 correctly reads "Land O Lakes salted butter" from the label — noticeably better than the base model. But it loses granular detail like size and variant for the milk and chicken.

2b. Image-Text Matching (ITM)

BLIP's ITM head is more directly useful for mismatch detection. Rather than generating text, it outputs a probability score between 0 and 1 representing how well an image and text match — directly interpretable without calibration.

from transformers import BlipProcessor, BlipForImageTextRetrieval

def compute_blip_itm_score(configs: Config, image_url: str, description: str) -> float:
    device = "mps" if torch.backends.mps.is_available() else "cpu"
    processor = BlipProcessor.from_pretrained(configs.blip_itm_model)
    model = BlipForImageTextRetrieval.from_pretrained(configs.blip_itm_model).to(device)

    img = fetch_image_from_url(image_url)
    inputs = processor(images=img, text=description, return_tensors="pt").to(device)

    with torch.no_grad():
        outputs = model(**inputs, use_itm_head=True)
        score = torch.nn.functional.softmax(outputs.itm_score, dim=1)[:, 1].item()

    return score

Description	ITM Score	Label
Correct (Prairie Farms milk)	0.9998	Likely match
Wrong brand (Organic Valley)	0.7872	Likely match
Wrong size (two quarts)	0.9996	Likely match
Obvious mismatch (cat playing)	0.0000	Likely mismatch

ITM is excellent at catching obvious mismatches. However, wrong brand and wrong size both score very high — the model is too lenient for fine-grained product matching. A wrong brand gets a 0.79 ITM score, which would pass as a likely match in a marketplace context.

This revealed a fundamental limitation: both CLIP and BLIP treat the description as a whole. Neither one systematically checks individual product attributes like brand, size, and variant in isolation.

The Lesson from Phase 1 and 2

CLIP and BLIP are powerful general-purpose models, but neither was designed to answer the specific question a marketplace needs: "Does the brand in the image match the brand in the description? Does the size match?"

What is needed is not a similarity score — it is a structured, per-feature comparison. That insight drives the approach in Part 2.

Why It Matters

Product content quality is not a niche concern. Research consistently links inconsistencies between images and text to reduced customer confidence and lower purchase intent [4]. Platforms that invest in automated content quality tooling see measurable improvements in conversion rates, return rates, and seller accountability.

An automated mismatch detection pipeline does not replace human review — it scales it. By surfacing the listings most likely to be wrong, it directs human attention to where it matters most, at a fraction of the cost of manual auditing.

What's Next

In Part 2, I replace free-form similarity scoring with a feature dictionary approach:

OCR (Optical Character Recognition) reads brand names, sizes, and variants directly from the product label as text
VQA (Visual Question Answering) answers targeted questions about visual features the OCR cannot read (container type, product category) and acts as a fallback when OCR misses stylised fonts or logos
A hard veto on all features ensures any single wrong attribute flags the listing — because in a marketplace, a wrong size is just as damaging as a wrong product

→ Continue to Part 2: OCR + VQA

Code and Experiments

All code and Jupyter notebooks for this project are available on GitHub:
github.com/ebiarian/product-image-description-alignment

References

Buzzin Content. 91.3% of E-Commerce Brands Failed to Address Real Customer Pain Points. https://www.buzzincontent.com/story/91-3-e-commerce-brands-failed-to-address-real-customer-pain-points-in-pdp-content-odn-report/
Besedo. The Effects of Fraud and Poor Content Quality on Online Marketplaces. https://besedo.com/blog/report-the-effects-of-fraud-and-poor-content-quality-on-online-marketplaces/
Perpetua Help. Missing or Incorrect Product Images. https://help.perpetua.io/en/articles/4626446-missing-or-incorrect-product-images
ScienceDirect. Inconsistencies between images and text in online shopping. https://www.sciencedirect.com/science/article/abs/pii/S0969698917307142