I'll be honest about where this started. I could get a vision model to look at a photo and spit back a caption. I could get two pieces of text to embed into vectors and compute a similarity score between them. What I couldn't do, not really, was explain what happens when those two things almost agree but shouldn't.
That gap bothered me. So for my capstone at FlyRank, I picked a project that forced me to sit inside that gap instead of walking around it: build a system that matches blog posts to the right animal photo, and refuses to guess when it isn't sure. A red fox post should get a red fox photo. A gray wolf photo should never sneak onto it, even if the two animals look close enough to fool a similarity score.
This is what that actually looked like, including the part where a coyote almost got past me.
What I was working with
About 50 images across five species that all kind of resemble each other if you're not paying close attention: red fox, gray wolf, coyote, Siberian husky, German shepherd. I picked those on purpose. Fox versus penguin versus shark would have been trivial, any system clears that bar. I wanted animals genuinely hard to tell apart, hard enough that even the model doing the classifying would sometimes hedge.
The pipeline, roughly: a vision model looks at each image and produces structured tags, subject, category, visible attributes, a caption, a confidence score. Image captions and post text both get embedded into the same vector space. Images get ranked by similarity to the post. And before anything gets suggested to a human, it has to survive a guard that's allowed to veto the top result entirely.
That last piece is the one I actually care about.
Why ranking alone wasn't enough
I wrote a test post. "Amazing shot of a wolf pack howling at night." Ran it through the ranker. Wolves came back on top, which is exactly what should happen. Then I kept scrolling through the rest of the results and found this sitting in the middle of them:
coyote_002.jpg | similarity: 0.6360 | REJECTED
reason: subject mismatch: post implies 'gray wolf', image is 'coyote'
A coyote photo scored 0.636 against my wolf post. The actual wolves in that same batch scored between 0.634 and 0.642. That coyote wasn't some outlier sitting off in the noise somewhere, it was sitting right inside the cluster of correct answers. A similarity only system would have handed it back with total confidence, and there would have been nothing in the output that looked wrong. High score, clean looking match, wrong animal.
I didn't sit down and design that test case to prove a point. I was just poking at my own system out of curiosity, and it handed me the exact proof I needed.
The guard itself checks two things. Is the similarity score above a threshold, and does the image's actual subject match what the post seems to be about. Fail either one and the pairing gets rejected with a plain English reason attached, not a silent no.
The part nobody warns you about
The AI logic took less time than getting the AI providers to cooperate.
I started with OpenAI and hit a billing wall almost immediately, there isn't really a free tier for vision anymore. Switched to Gemini and got a limit: 0 quota error on one model and a 20 requests a day cap on another, which meant my batch job classifying 49 images ended up spread across three separate days because I kept running out of quota partway through a run. Tried Groq next since I'd used it before on a different project, only to find my account had zero vision capable models on it at all.
None of that was a code problem. It just ate far more hours than the actual guard logic did. What came out of it was a batch script with retries and a check that skips anything already classified, built specifically so I could stop the process, walk away, come back the next day, and pick up exactly where the quota had cut me off, without wasting API calls reclassifying images I already had results for.
Small thing. It was also the difference between a script I could actually walk away from and one I had to babysit.
What actually shipped
Vision classification with a validated schema, where low confidence results get flagged for human review instead of quietly trusted. Batch processing with retries and per call cost tracking, the whole project ended up costing something like $0.0002 total. Semantic matching that handles paraphrasing, a post written with "Vulpes vulpes" instead of "red fox" still correctly pulls up real fox photos, zero words shared between them. The mismatch guard itself, proven on the coyote case above. A small review API to approve or reject suggested pairings. And 100% top-1 precision on a labeled evaluation set.
I had some momentum left after all of that, so I built a React frontend on top of it. Type a post, click a button, watch the match appear. Watch a rejected candidate show up grayed out with the reason sitting right underneath it, instead of just disappearing silently.
What I'd tell someone starting this same project
Go looking for the case that breaks your system, don't wait for it to find you. I didn't design the coyote scoring 0.636 scenario, I found it by throwing an ambiguous post at my own pipeline out of idle curiosity. If I'd only ever tested the easy cases, a clean fox post, a clean wolf post, I would have shipped something that looked finished and wasn't.
And leave room in your timeline for the parts that have nothing to do with the skill you're actually trying to practice. Three different providers, three different ways of falling over, taught me more about reading error messages slowly and not trusting my own first assumption than the machine learning side of this project ever did. I once decided a completely valid API key had to be broken, based on nothing but a prefix I didn't recognize. I was wrong, and testing it directly instead of arguing with myself would have saved me twenty minutes.
None of this was hard in the sense of requiring brilliance. It was hard in the sense of requiring patience, and a willingness to actually sit with why something broke instead of guessing at fixes until the error went away.
The repo's up on GitHub if you want to see the schema, the guard logic, and how the whole pipeline fits together: [https://github.com/YassineS99/semantic-image-matcher.git]
Next up, I want to grow the corpus and see whether the guard holds up on species that don't look nearly as similar as foxes and wolves do. If you've built something with a similar "refuse instead of guess" layer in it, I'd genuinely like to hear where yours drew the line.
Top comments (1)
The coyote at 0.636 sitting inside a wolf cluster of 0.634–0.642 is the most useful number in this post, and I think it proves something a bit stronger than "ranking alone wasn't enough."
There is no threshold that fixes it. Any cutoff at or below 0.636 lets the coyote through; any cutoff above it starts rejecting real wolves at 0.634. The two distributions overlap, so at that operating point the decision simply isn't available in that signal — not at any threshold. Which is exactly why your subject check works: not because it's stricter, but because it's a different axis. Tightening the number would only have traded one error for the other.
I hit the same shape from the opposite direction recently, measuring tree-to-tree correlation in a Random Forest. The setting that scored best on the proxy I was optimising (lowest correlation, 0.248) turned out to be the worst model on held-out data, by 21 points of R2. The proxy was real and measured correctly. It just wasn't the goal.
One thing I'd be curious about: do the vision model's own confidence scores correlate with what your guard ends up rejecting? If it was already hedging on that coyote, you may have a cheap second axis sitting in data you're already collecting.