We asked 5 AI models the same 24 questions. They agreed on the facts — and split on almost every judgment call.

MultiO — Sat, 07 Nov 2026 02:49:16 +0000

We ran a small experiment. We took 24 questions — six each on pricing/strategy, factual questions with a trap, coding approach, and recommendations with tradeoffs — and sent each one, unchanged, to five models: GPT, Claude, Gemini, Grok and Llama (mid-tier model from each family). Then we compared where they actually disagreed in substance, not style.

The headline: on 8 of 24 questions (33%) the models materially agreed. On 16 of 24 (67%) there was at least one material divergence — 52 in total. And on 15 of 24 (63%), one model stood completely alone: a claim no other model made, or a dissent against everyone else.

The splits were not random. Factual questions were near-unanimous (0.33 divergences per question). Judgment calls split hard: recommendations averaged 3.17 divergences per question, coding defaults 2.67, pricing 2.50.

Two examples stuck with us:

Asked whether an indie app earning $3,000/month from a $30 one-time unlock should switch to $5/month, one model said an unconditional yes, two said no unless retention cleared six months — and only one priced the transition: you'd need ~600 active subscribers just to match the old revenue, with a dip and backlash risk on the way. That lone line was the most useful one in the set.
Asked about the spiders-in-sleep myth, every model correctly called it false — but only one added a confident origin story for the statistic. That origin story is itself unverified; it traces to a source that has never been shown to exist. The lone claim was the most interesting line in its set, and the only wrong one.

That is the pattern across the study: of 14 sole-source claims, 8 (57%) held up as the highest-value line in their answer set, and 3 were wrong or invented. The outlier is sometimes the product and sometimes the trap — and you cannot tell which from a single model's answer, because a single model never flags its own lone claims.

Control check: we re-ran two questions three times each through one model. It never changed its verdict — the disagreement lives between models, not inside one model's sampling.

Full method, caveats (including the judge being a panel member, and a memory-bleed incident we found and disclosed) and all 24 questions: https://multio.ai/research/model-divergence

Disclosure: I build MultiO (multio.ai), the multi-model app this study ran on — one subscription, every top AI model, and Auto sends each message to the most cost-efficient model that can handle it. The study is the reason we built Compare and Council: if models split on judgment calls two times out of three, asking more than one is not a luxury feature.

— @getmultio / hello@multio.ai

DEV Community: MultiO

We asked 5 AI models the same 24 questions. They agreed on the facts — and split on almost every judgment call.