Two new benchmarks make an uncomfortable case: the strong scores vision-language models post on standard tests are partly an averaging illusion, and the models are still surprisingly bad at reliably perceiving fine visual detail. One, called PerceptionRubrics, replaces holistic scoring with a strict 'gated' method that fails a model outright when it misses an essential fact -- and in doing so exposes a persistent gap between open and proprietary systems that ordinary benchmarks paper over.
Key facts
- The method: PerceptionRubrics grades with 'must-right' and 'easy-wrong' facts and a gated penalty -- miss a mandatory fact and the score drops sharply.
- The finding: a persistent ~8% perception gap between open-source and proprietary models, even at similar reasoning ability.
- The companion fix: PixelEyes (arXiv:2607.00115) hands precise localization to a specialized tool instead of the model.
- Primary source: PerceptionRubrics, arXiv:2606.28322.
A vision-language model takes an image plus a question and answers in words -- describing a scene, reading a chart, spotting an object. The standard way to grade one is to average partial credit across many sub-questions, which rewards a model for getting the easy, common details right. PerceptionRubrics argues that average hides the failures that matter. It scores against 'must-right' essential facts and 'easy-wrong' fine-grained traps, and gates the result: blow a mandatory fact and you take a sharp binary penalty, the way a single critical error should sink an answer even if everything else is fine. Under that stricter lens, models that looked strong turn out to be perceptually brittle -- good at fragments, bad at satisfying strict all-or-nothing constraints -- and a stubborn 8% perception gap opens between open-source and proprietary models even when their reasoning is comparable. The difference, in other words, isn't how they think; it's how well they actually see.
The second paper, PixelEyes, tackles why that seeing is so fragile and offers an architectural fix. Today's all-in-one models try to reason about an image and pinpoint exact locations within it at the same time, and they're bad at the second job -- a model that can't localize a target keeps triggering extra reasoning turns to compensate, bloating the process with redundant work. PixelEyes decouples the two: it delegates fine-grained localization to a specialized perception tool while the language model keeps the high-level reasoning. A modular 'eye plus brain' beats one model doing both, cutting the wasted trajectories. It's the same lesson echoing across this week's research -- reliability comes from wrapping a model in the right specialized tools, not asking it to do everything at once, exactly as biology agents needed a deterministic lookup tool to stop failing at retrieval.
The caveat is that these are new, sharply designed benchmarks, and a metric tuned to expose brittleness will, by construction, make models look brittle; the 8% gap is a finding about these tasks, not a universal verdict. But the direction is consistent and useful: as vision models get folded into agents that act on what they see, 'looks right on average' is not good enough, and tests that punish the critical miss are how the field will find the gaps before they ship. Read PerceptionRubrics and PixelEyes.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)