This is Part 2 of a two-part series. Part 1 covered the problem, the CLIP baseline, and BLIP. This part covers the OCR + VQA pipeline that replaced them for fine-grained product feature matching.
The Problem with Scores
At the end of Part 1, the conclusion was clear: CLIP and BLIP are good at catching obvious mismatches but fail at the fine-grained ones that matter most in a marketplace. A wrong brand scores 0.79 on BLIP ITM — comfortably inside "Likely match" territory. A wrong size is nearly invisible to both models.
The reason is architectural. Both CLIP and BLIP compare an image and a description as whole units. They never ask: does the brand in the image match the brand in the description? They encode everything together and produce a single score. That score cannot tell you which attribute caused a mismatch.
What a marketplace actually needs is not a similarity score. It needs answers to specific questions:
- Is this the right brand?
- Is this the right size?
- Is this the right variant?
That insight led to a completely different approach.
The Three Products Used in This Article
All experiments in this article are run against three real grocery products — chosen to represent different label types, packaging materials, and mismatch challenges:
- Prairie Farms Whole Milk (1 quart carton) — a text-heavy label with a stylised brand logo, nutritional facts on the back, and a clear front panel. Useful for testing brand and variant detection.
- Amazon Fresh Boneless Skinless Chicken Breast (30 oz plastic container) — minimal label text, brand printed on a sticker. Tests whether the model can identify both the brand and the specific cut.
-
Land O Lakes Salted Butter (8 oz paper wrapper) — compact packaging with multiple size representations printed on the label (
8 oz,226.8 g,4 half sticks). Tests size matching edge cases.
The Feature Dictionary Approach
Instead of scoring image against description holistically, the idea is to define a feature dictionary — a structured set of attributes that any product listing should have — and verify each one independently.
description_features = {
"product_type": "milk",
"brand": "prairie farms",
"size": ["1 quart", "946ml", "1 qt"], # multiple equivalent representations
"variant": "whole",
"container": "carton",
}
For each feature, the system extracts the corresponding value from the product image and compares it against the description. If any single feature fails — the listing is flagged.
Two tools do the extraction, each handling what the other cannot:
Tool 1: OCR for Text Features
OCR (Optical Character Recognition) reads printed text directly from an image pixel by pixel and converts it into a string. For product labels, the most important information — brand name, weight, variant — is printed text. Reading it directly is faster and more accurate than asking a vision model to interpret it.
We use EasyOCR (pip install easyocr), a Python library that runs locally with no API calls required.
import easyocr
import numpy as np
from src.utils import fetch_image_from_url
def extract_ocr_text(image_url: str, min_confidence: float = 0.4) -> str:
img = fetch_image_from_url(image_url)
img_array = np.array(img)
reader = easyocr.Reader(["en"], gpu=False, verbose=False)
results = reader.readtext(img_array)
texts = [text for _, text, conf in results if conf >= min_confidence]
return " ".join(texts).lower()
OCR on the Land O Lakes butter label returns:
jarmer-owned landolakes since 1921 tbsp 4 half sticks butter half net wt 8oz (226.8 g) salted
"land o lakes", "butter", "8oz", and "salted" are all there.
The space stripping trick
OCR commonly reads "8oz" as a single token while the description says "8 oz" with a space. A naive substring match would fail. The fix is to strip all spaces before comparing:
ocr_found = any(
d.replace(" ", "") in ocr_text.replace(" ", "")
for d in desc_norms
)
Multiple equivalent representations
Size values are inconsistent across listings. A butter listing might say "8 oz", "226.8 g", or "4 half sticks" — all referring to the same product. Requiring an exact match would cause false negatives. The solution is to accept a list of equivalent representations for any feature:
"size": ["8 oz", "226.8 g", "4 half sticks"]
If OCR finds any one of them, the feature passes.
Tool 2: Moondream2 VQA as Fallback
OCR has two hard limits. First, it cannot read stylised logos and custom brand fonts — "Prairie Farms" rendered as a decorative script may return garbled characters or nothing at all. Second, some features are not printed as plain text at all — "container type" is not written on the label; it is a visual property the model must infer.
For these cases, the fallback is VQA (Visual Question Answering) using Moondream2 (1.86B parameters, vikhyatk/moondream2).
VQA models take an image and a natural language question and return a short answer. Unlike captioning — which describes everything and may miss the specific detail you care about — VQA is targeted and controllable.
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"vikhyatk/moondream2",
revision="2025-01-09",
trust_remote_code=True,
torch_dtype=torch.float16,
).to(device)
tokenizer = AutoTokenizer.from_pretrained("vikhyatk/moondream2", revision="2025-01-09")
enc_image = model.encode_image(image) # encode once, ask many questions
answer = model.answer_question(enc_image, "What brand name is shown on this product?", tokenizer)
Why Moondream2?
| Model | Size | Why chosen |
|---|---|---|
blip-vqa-base |
~400MB | Weak instruction-following, poor on product labels |
Salesforce/blip2-opt-2.7b |
~5.5GB | No VQA head, generative only |
vikhyatk/moondream2 |
~4GB | Strong instruction-following, efficient on M2, handles logos well |
Moondream2 runs in float16 on Apple M2 MPS and uses approximately 4 GB of unified memory — comfortable on a 16 GB machine.
Constrained prompts
Open-ended questions invite unpredictable answers. Constrained prompts force the model to choose from a known vocabulary, which reduces hallucination:
FEATURE_QUESTIONS = {
"product_type": "Is this milk, coffee, butter, chicken, meat, bread, juice, a beverage, snack, or cleaning product?",
"container": "Is this product in a bottle, can, carton, container, canister, bag, or box?",
"brand": "What brand name is shown on this product?",
"size": "What is the size or quantity? Find a number with these words: oz, quart, gram.",
"variant": "What variant or flavor is this? For example: whole, 2%, dark roast, classic roast.",
}
Putting It Together: OCR First, VQA Fallback
The routing logic is simple:
For text features (brand, size, variant, product_type):
1. Try OCR → if found, done
2. If not found → ask Moondream2
For visual features (container):
→ Ask Moondream2 directly (not printed as text)
def compare_features(image_features: dict, description_features: dict) -> dict:
ocr_text = image_features.get("_ocr_text", "")
for feature, desc_val in description_features.items():
desc_norms = [v.strip().lower() for v in desc_val] if isinstance(desc_val, list) else [desc_val.lower()]
if feature in TEXT_FEATURES:
ocr_found = any(d.replace(" ", "") in ocr_text.replace(" ", "") for d in desc_norms)
if ocr_found:
matched, source, img_answer = True, "OCR", "found in OCR"
else:
vqa_val = image_features.get(f"_vqa_{feature}", "").lower()
matched = any(vqa_val in d or d in vqa_val for d in desc_norms)
source, img_answer = "VQA fallback", vqa_val
else:
vqa_val = image_features.get(f"_vqa_{feature}", "").lower()
matched = any(vqa_val in d or d in vqa_val for d in desc_norms)
source, img_answer = "VQA", vqa_val
Results on Three Products
| Product | brand | product_type | size | variant | container | Overall |
|---|---|---|---|---|---|---|
| Prairie Farms Whole Milk | ✓ VQA | ✓ OCR | ✓ OCR | ✓ OCR | ✓ VQA | ✅ Match |
| Amazon Fresh Chicken Breast | ✓ OCR | ✓ OCR | ✓ OCR | ✓ OCR | ✓ VQA | ✅ Match |
| Land O Lakes Salted Butter | ✓ OCR | ✓ OCR | ✓ OCR | ✓ OCR | ✓ VQA | ✅ Match |
All correct descriptions matched. Wrong brand, wrong size, and wrong variant correctly triggered mismatches across all three products.
Every Feature Is Critical
An early version of the labelling logic used a ratio: if 75% of features match, the listing passes. This is wrong for a marketplace.
A customer ordering 8 oz of butter who receives 16 oz has received the wrong product. A boneless chicken breast described as bone-in is a real fulfilment failure. There is no threshold at which a wrong size becomes acceptable.
The final labelling logic is a hard veto:
CRITICAL_FEATURES = {"brand", "product_type", "size", "variant", "container"}
def overall_label(feature_results: dict) -> str:
for feature in CRITICAL_FEATURES:
if feature in feature_results and not feature_results[feature]["match"]:
return "Likely mismatch" # immediate, no exceptions
return "Likely match"
Any single feature failure returns "Likely mismatch" regardless of what else matched.
Limitations
This approach is meaningfully better than CLIP or BLIP for fine-grained product matching, but several real-world challenges remain:
Image angle: OCR reads only what is visible. If the listing image shows the nutrition facts panel rather than the front label, brand and variant text will be missed entirely. The fix — accepted multiple image URLs per product and merge OCR results — is left for future work.
Image quality: Blurry or compressed images reduce OCR accuracy and may cause Moondream2 to hallucinate answers for features it cannot clearly see.
Prompt coverage: The current VQA prompts cover a handful of product categories. In a real marketplace with thousands of categories, prompts would need to be carefully engineered per category — broader option lists, context-aware phrasing, potentially chain-of-thought reasoning. The current prompts are a proof of concept, not production-ready.
Semantic matching: The comparison logic uses substring matching after space normalisation. "1 qt" and "one quart" are not matched even though they are equivalent. Embedding-based semantic matching would handle this but adds complexity.
Takeaways
The key insight from this project is that the right framing matters more than the right model. CLIP and BLIP are powerful, but they answer the wrong question — "how similar are these overall?" — when the marketplace needs answers to specific questions: wrong brand? wrong size? wrong variant?
Switching to a feature dictionary approach with OCR and VQA fallback made the system interpretable (you can see exactly which feature failed), debuggable (you can inspect the OCR text and VQA answers directly), and strict (a single wrong feature is enough to flag a listing).
The combination of EasyOCR for printed text and Moondream2 for visual reasoning is not perfect — but it is a meaningful step toward a system that could be useful in production.
Code and Experiments
All code, notebooks, and experiment results are available on GitHub:
github.com/ebiarian/product-image-description-alignment
← Back to Part 1: CLIP & BLIP
Top comments (0)