I Ran the Same Photo Through 3 AI Models. They All Disagreed. Here's the Bug.

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

The Bug That Taught My AI to Say "I'm Not Sure"

The Project

I'm building AURA AI — a personal presentation, grooming, fitness, and lifestyle coaching app that analyzes a user's photos and lifestyle info to generate a personalized improvement plan. Before writing a single line of production code, I was stress-testing the "brain" of the app: the prompt that turns a few selfies into a structured, safe, evidence-based report.

To do that, I ran the exact same test — same photos, same lifestyle data, same instructions — through three different AI models: Claude, Gemini, and DeepSeek.

The Bug

The first few runs looked great individually. Each model produced a clean, confident report: skin texture notes, hair analysis, posture observations, a prioritized "what to improve" list.

But when I lined the three reports up side by side, something was off. The same input photos were producing different conclusions depending on which model I asked — and sometimes even between two runs on the same model. One report would call a shadow under someone's eyes "visible fatigue lines." Another, using a photo taken in different lighting, wouldn't mention it at all.

The models weren't malfunctioning. They were doing exactly what I told them: describe what's visible in the photo. The actual bug was upstream — in my inputs. My test photos varied wildly in lighting, angle, and distance, and the AI had no way to tell the difference between "this is a real feature of the person" and "this is a shadow because the bedroom light was warm-toned." It was confidently reporting photographic noise as fact.

That's a genuinely dangerous bug for an app that's supposed to give people advice about their own face and body — confidently-wrong output is worse than no output.

How I Found It

I built what I ended up calling a Problem Detection Engine: instead of trusting the AI's first answer, I made it run a self-audit checklist after every report, forcing it to answer PASS/FAIL on questions like:

• Did I treat an inference as an observation?
• Did I confuse a lighting artifact with a genuine feature?
• Did I compare this photo to another under inconsistent conditions?

Running this audit across all three models is what surfaced the pattern. It wasn't one bad output — it was a systemic gap in how any model handles uncontrolled input photos.

The Fix

I fixed it in two layers, because patching the prompt alone wasn't enough:

1. Constrain the input, not just the output.
I wrote an explicit photo capture protocol into the pipeline — fixed camera distance (1.2–1.5m), eye-level angle, tripod-stabilized, consistent lighting — and replaced my inconsistent test set with a controlled one. Garbage in, garbage out applies just as much to vision prompts as it does to databases.

2. Make uncertainty a first-class output.
Instead of a binary "this is a feature," I added a confidence system: every observation now gets tagged LOW / MEDIUM / HIGH confidence, and anything with unreliable lighting or angle gets explicitly flagged as "not determinable" rather than guessed at. If the evidence is bad, the AI now says so instead of filling the gap with a confident-sounding guess.

The Code

Here's a simplified version of the confidence-tagging rule I added to the prompt, so the model flags uncertain observations instead of guessing:

For every visual observation, output:
{
  "observation": "<what you see>",
  "confidence": "LOW" | "MEDIUM" | "HIGH",
  "reason": "<why this confidence level>"
}

Rules:
- HIGH: clearly visible under consistent, even lighting
- MEDIUM: visible but lighting/angle introduces some ambiguity
- LOW: could plausibly be a lighting artifact, shadow, or angle distortion

If confidence is LOW, do not state the observation as fact.
Instead output: "Not determinable from this image — retake under
even, front-facing light."

This one rule is what turned "confidently wrong" into "honestly uncertain" — the exact fix the self-audit below confirms.

The Result

Re-running the same three models on the corrected photo set collapsed the inconsistency almost entirely — the reports agreed on what was actually observable, and openly disagreed (flagged as uncertain) on the handful of things that genuinely were ambiguous, instead of silently picking different answers.

What I Took Away From This

The most dangerous bugs in an AI product aren't the ones that crash — they're the ones that answer confidently and wrong. Cross-testing the same prompt across multiple models turned out to be a great debugging technique on its own: if three independent systems disagree on the same input, the bug usually isn't in any one of them — it's in what you're feeding them.

I'm still pre-code on the full AURA AI build, but this round of prompt-level bug hunting is exactly what's shaping how I design the real pipeline: validate the input before you ever trust the output.