The latest articles on DEV Community by Tomer Bar-Meir (Tom) (@tomerbarm). https://dev.to/tomerbarm https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4144318%2F97023197-3e2c-422a-b1f4-5c7970199674.jpg https://dev.to/tomerbarm en Tomer Bar-Meir (Tom) Sat, 26 Sep 2026 12:42:43 +0000 https://dev.to/tomerbarm/why-general-purpose-vision-ai-cant-tell-which-way-your-product-photo-is-rotated-15jj https://dev.to/tomerbarm/why-general-purpose-vision-ai-cant-tell-which-way-your-product-photo-is-rotated-15jj <p>Every product-photo pipeline we've built makes one quiet assumption: the photo is right-side up. It almost never is, and when it isn't, everything downstream degrades a little without anyone noticing why.</p> <p>We assumed a modern vision-language model would just handle this. It doesn't. An independent, peer-reviewed paper (RotBench, EACL 2026, arXiv:2508.13968) found that GPT-5, o3, and Gemini-2.5-Pro all reliably confuse a 90 degree rotation with a 270 degree one -- not a small effect, roughly half of their errors fall in that one direction. We ran our own comparison against ChatGPT, Gemini, and Claude on real product photos and found the exact same failure mode. In our Claude evaluation specifically, this single confusion accounted for 37% of every error the model made.</p> <h2> What we built instead </h2> <p>A domain-specialized system, not a prompt or a wrapper around an existing model:</p> <ul> <li>One model finds the nearest clean 90 degree turn (a 4-class classifier)</li> <li>A second model reads the precise leftover tilt (fine-angle regression, 360-bin circular target)</li> <li>A third, small trained model arbitrates between them when they disagree, using both models' own confidence signals</li> </ul> <p>The two base models have genuinely complementary blind spots -- one is excellent on clean, easy photos and weak on heavy tilt, the other is the reverse. Combining them beats either alone by a real, statistically significant margin, verified via photo-cluster bootstrap (not a naive per-image confidence interval, which would overstate significance given correlated crops from the same source photo).</p> <h2> The numbers </h2> <p>Measured on a large, individually hand-verified pool of real product photos, not a synthetic benchmark:</p> <ul> <li> <strong>99.02%</strong> accuracy on a clean quarter-turn</li> <li> <strong>96.67%</strong> across the full 360 degree rotation range</li> <li> <strong>84.65%</strong> on the single hardest case -- a tilt exactly halfway between two clean angles</li> </ul> <p>Against ChatGPT specifically, on a consolidated 394-photo sample: a <strong>+15.2 percentage-point lead</strong> on quadrant accuracy (91.1% vs. 75.9%), 95% CI [+10.4pp, +20.0pp] via photo-cluster bootstrap. Across every assistant tested, the same 90/270 confusion RotBench documented showed up again, independently, in our own data. Our system shows no equivalent pattern anywhere in its results.</p> <h2> What we don't hide </h2> <p>The full paper discloses what we'd want to know before trusting a number like this ourselves: a genuine tilt right at the halfway point between two angles is still the hardest case (84.65%, not 99%); the system hasn't been tested on photos with more than one valid "upright" (a flat-lay, a floating render); and one internal component's raw confidence turned out to be structurally uninformative at the high end -- confidently wrong almost as often as confidently right -- which we root-caused and partially mitigated with a same-model consistency check, not swept under the rug.</p> <p>Full paper, methodology, and dataset construction: <a href="https://doi.org/10.5281/zenodo.22975679" rel="noopener noreferrer">https://doi.org/10.5281/zenodo.22975679</a></p> <p>Happy to go deeper on the arbitration layer, the calibration issue, or the eval methodology in the comments.</p> machinelearning computervision ai showdev