A number named confidence_score can look more authoritative than the experiment behind it. In a marketplace filter, a high image-text similarity score does not prove that the seller is honest, that an appliance works or that the photographed item is available.
Eşya Bul’s public verifier provides a concrete example. It compares a listing image with a small set of descriptions using CLIP, then keeps the listing if the best positive score exceeds 0.85. Reading the code helps separate a useful visual filtering operation from a much broader trust claim.
Identify the actual question
The inspected revision loads openai/clip-vit-base-patch32. It builds candidate product groups from an existing category tag and title keywords. For a refrigerator, the positive description is a photograph of a refrigerator; alternatives include parts, repair services and a broken fridge.
For each candidate group, the code sends the image and descriptions to the model, applies softmax to the image logits and takes the score of the first, positive description. If more than one group is considered, it keeps the group with the highest positive score. The retained result includes that group and the score.
The narrow question is: which of these supplied descriptions best matches this photograph? That can help remove a replacement handle from a search for a complete refrigerator. It cannot establish ownership, working condition, delivery reliability or payment safety. Those require different evidence.
Category selection matters too. An inaccurate title or tag can make the model compare the image against the wrong alternatives. There is no general-purpose verification step hiding behind the word “verified” in the output filename.
Keep the denominator in view
Softmax normalizes a set of logits into shares that sum to one. The denominator depends on the candidates included in that particular comparison. Add a strong alternative and the positive candidate’s share can fall even when its own logit stays unchanged.
Here is a self-contained illustration:
from math import exp
def positive_share(logits):
top = max(logits)
weights = [exp(value - top) for value in logits]
return weights[0] / sum(weights)
print(round(positive_share([4, 1, 0]), 3)) # 0.936
print(round(positive_share([4, 1, 0, 4]), 3)) # 0.484
These are invented logits for an arithmetic demonstration, not model outputs or a benchmark. The example shows why a threshold cannot be interpreted without its candidate design. It does not show how accurate CLIP is on refrigerator listings.
There is another comparison to handle carefully: the verifier selects the highest positive score across product groups with different alternatives. Each score was normalized within its own group. A larger number in one group is not automatically a calibrated probability comparable to every other group.
OpenAI’s model card discusses sensitivity to category design and the need to evaluate a model in its intended context. Fine-grained distinctions and non-English use also have limitations. In this implementation, Turkish title keywords select groups, while the CLIP descriptions are English. Those two stages deserve separate error analysis.
Evaluate the decision the product actually needs
Start with a label definition small enough for humans to apply consistently. “The image shows a complete refrigerator rather than a replacement part” is much more testable than “this is a safe listing.”
| Outcome | Example | Why it needs a separate label |
|---|---|---|
| Visual match | A complete appliance | Find useful listings rejected by the filter |
| Visual mismatch | A handle or repair advertisement | Find unwanted listings accepted by the filter |
| Ambiguous | A collage containing an appliance and loose parts | Preserve uncertainty rather than force a clean label |
| Technical failure | An image cannot be downloaded | Separate network failures from model decisions |
The current script skips missing images, failed downloads and rejected classifications before writing its final list. A shorter output list therefore cannot establish better filtering. Add explicit outcome reasons before interpreting the disappearance of an item as model success.
For an evaluation, keep near-duplicate photographs from the same listing on one side of the development/evaluation split. Record disagreements between human reviewers. Compare false accepts and false rejects at several thresholds, broken down by product group. Freeze the candidate descriptions for each reported run so that another person can reproduce the comparison.
The appropriate threshold depends on the next action. A shortlist reviewed by a person may tolerate more false accepts to avoid hiding useful items. A decision that triggers an automated action can have a different error cost. No threshold is justified merely because its decimal representation resembles a familiar percentage.
The defensible claim supported by this code is preliminary filtering for visual category consistency. This article does not report a new benchmark or establish a reliable fraud detector.
Sources and original version
- Original article on kaahn.com
- Eşya Bul verifier at the reviewed revision
- OpenAI CLIP model card at the reviewed revision
- Learning Transferable Visual Models From Natural Language Supervision
AI disclosure: This article was drafted by an AI agent from the linked code and documentation. The softmax example is illustrative; no new model evaluation was performed for this article.
Top comments (0)