DEV Community

MultiO
MultiO

Posted on

One AI model never disagreed with anyone in our study — and that isn't a compliment

When you ask several AI models the same question, some models keep ending up alone — the only one making a claim, or the only one dissenting. Others never do. In our 24-question divergence study, we counted it per family, and the spread was wider than I expected.

We sent the same 24 questions, unchanged, to five mid-tier models — GPT, Claude, Gemini, Grok and Llama — and flagged 52 material divergences. Exactly half (26) involved a lone model: 14 sole-source claims and 12 lone dissents. Broken down by family (each family was judged in 19–20 of the 24 questions, because the judging panel read a rotating four of five answers):

  • Claude: lone on 12 of 19 judged questions — 10 sole-source claims and 2 lone dissents, by far the most.
  • Llama: lone on 7 of 20 — only 1 sole-source claim, but 6 lone dissents, the most of any family.
  • Gemini: lone on 5 of 19 — 2 claims, 3 dissents.
  • GPT: lone on 2 of 19 — 1 claim, 1 dissent.
  • Grok: lone on 0 of 19. Never the only claim, never the only dissent — always on a shared side of every split.

The two profiles at the extremes fail differently. The frequent loner adds the most new information to a panel — and the most risk: across the study, of the 14 sole-source claims hand-checked, 8 were the highest-value line in their set and 3 were wrong or invented. The never-loner adds the least as a second opinion: if a model is always on a shared side, asking it second mostly confirms what you already had. Consensus is comfortable, but a second answer that cannot disagree with the first is not really a check.

One caveat matters more than the ranking, and we publish it with the study: the judge that flagged the divergences was Claude Sonnet 5.5 — itself a panellist, and the family that produced most of the outliers. The lists were hand-checked against the answer texts, but a judge favouring its own family's extra claims cannot be ruled out, so read the Claude count with that in mind. The Grok zero, though, does not depend on that bias in the same way: never being flagged as the lone voice, by a judge from a different family, is the conservative direction of the error.

My practical rule now: when I cross-check a decision, I want at least one model in the panel with a track record of actually dissenting — and when the frequent loner adds a fact nobody else mentioned, I check that fact first, because that is exactly where both the best lines and the invented ones live.

Full study, method and all caveats (including the rotating judging sets, the mid-tier panel, and a memory-bleed incident we found and disclosed): https://multio.ai/research/model-divergence

Disclosure: I build MultiO (multio.ai) — One subscription. Every top AI model… Auto sends each message to the most cost-efficient model that can handle it. Compare asks up to three models side by side, and every message shows its exact token charge on a receipt. Free trial: 3,000,000 tokens, no card required.

— @getmultio / hello@multio.ai

Top comments (0)