DEV Community

MultiO
MultiO

Posted on

Recommendation questions are where AI models disagree most of all

If you ask an AI for a recommendation — which car, which tool, which plan — you are asking in exactly the category where models disagree most.

We sent the same 24 questions, unchanged, to five models — GPT, Claude, Gemini, Grok and Llama. Six were recommendations with tradeoffs. On factual questions the panel was near-unanimous (0.33 material divergences per question). On recommendations, they averaged 3.17 divergences per question — the highest of any category — and 5 of the 6 recommendation questions produced at least one material split.

Three recommendation examples from the study:

  1. Minivan vs 3-row SUV: most answers converged on sliding doors and step-in height. Only one model stated when its own recommendation flips — towing over about 3,500 lbs, real off-road or all-weather needs, and image. That condition is the part a buyer actually needs, and a single-model workflow would have missed it.
  2. Bakery commercial oven: every judged model said finance a commercial oven now, on capacity grounds. Only one raised the point that could void the whole comparison — some health inspectors and insurers will not accept domestic ovens for commercial production. It was also the only answer with implementation guidance: used or refurbished units, a lease-vs-loan comparison, and budgeting for hookups and ventilation.
  3. Remote team communication: three models picked async-first. One dissented and picked Slack-style chat as the main tool, arguing chat is already effectively asynchronous across time zones — a premise the other answers rejected. Same question, opposite operating model, presented as settled either way.

That is the pattern across the study: of 14 sole-source claims, hand-checked, 8 were the highest-value line in their set and 3 were wrong or invented. On a recommendation, the lone voice is sometimes the only answer that named the condition that changes the decision — and sometimes the one inventing a premise. You cannot tell which from a single answer, because a single model never flags that it is alone.

My practical rule now: one model is fine for a factual lookup; for a recommendation I will live with — a vehicle, a lease, a tool stack, a training plan — I look at where more than one model splits before I act. The disagreement is the information.

Full study, method and caveats (including that one judge was also a panel member, and a memory-bleed incident we found and disclosed): https://multio.ai/research/model-divergence

Disclosure: I build MultiO (multio.ai) — One subscription. Every top AI model… Auto sends each message to the most cost-efficient model that can handle it. Compare asks up to three models side by side, and every message shows its exact token charge on a receipt. Free trial: 3,000,000 tokens, no card required.

— @getmultio / hello@multio.ai

Top comments (1)

Collapse
 
aiapikey profile image
HAOHAOnovai •

The finding is solid and the metric measures the wrong thing for the use case. Divergences per question rewards models for differing, but what a buyer needs is the condition that flips the answer - your towing threshold, the inspector who will not accept a domestic oven. You show that twice and never count it.

Score each answer binary for "names its flip condition", then report the share of questions where at least one model names it against the share where none does. That is the number that says whether a panel beats a single model, and it is computable from the answers you already have.

One caveat: 6 recommendation items. 3.17 is directionally right; the category ranking needs more items before it is a ranking.