A while back I got annoyed at a specific genre of blog post: "we asked ChatGPT what the best CRM is and here's the answer." One screenshot, one run, treated as if the model holds a stable opinion. It doesn't. So I built a small harness to measure that instead of hand-waving about it.
The setup is boring on purpose. Eight models. Sixteen B2B software categories (CRM, project management, email marketing, that kind of thing). For each category I ask every model the same plain question: what is the single best tool here. One pick, no hedging allowed. I log the raw response, the parsed pick, the model, the timestamp, and the exact prompt into a JSONL file. Then I do it again next month.
Two things fell out that I did not fully expect.
First, across all sixteen categories the eight models never once agreed on the same tool. Not close-but-different. Zero unanimous picks out of sixteen. I assumed there would be at least a couple of categories where everyone converged on the obvious incumbent. Nope.
Second, and this is the one I keep chewing on: the models do not even agree with themselves. Ask the same model the same question in a fresh session and it swaps its own top pick around 74% of the time. Same model, same prompt, nothing changed but the session. Roughly three-in-four odds it contradicts what it told you yesterday.
Here is the part I want other people to break. I do not know how much of that self-disagreement is temperature and how much is genuine ranking instability. My instinct is that dropping temperature to 0 will not actually fix it, it will just hide the wobble behind a deterministic-looking facade while the underlying ranking stays mush. But I have not run that ablation cleanly yet, and I would rather someone who does eval work for a living tell me I am wrong.
The reason I am posting the method and not only the numbers: a "models disagree" claim is worthless if you cannot reproduce it. So everything is open. The per-run JSONL, the prompt text, the model list, all of it, DOI'd under CC-BY so you can cite it or tear it apart: https://data.deepsynthesis.org/. I re-run the whole thing monthly, which means you can watch a specific category drift over time rather than trusting a frozen screenshot from whenever the author happened to hit send.
A few implementation notes if you want to build your own version:
- Force a single pick. If you let the model give you a ranked list it will hedge and you lose the signal. One tool, no runner-ups, is the whole trick.
- Parse defensively. Models love to answer the question and then immediately undercut it with "but it depends on your needs." Strip that.
- Store the raw response next to the parsed pick. You will second-guess your parser and you will want the original text.
- Timestamp everything and never overwrite a run. The monthly diff is the actual product.
The uncomfortable takeaway for anyone doing AI-search or GEO work: if a model's pick is this unstable, "we rank first when you ask ChatGPT" is a coin that lands differently every flip. Optimizing for a single snapshot is optimizing for noise. Optimizing to show up across many runs and many models is the only thing that survives.
If you poke at the data and find a category where the parser is wrong, or a model I should add, tell me. That is the point of putting it out in the open.
Top comments (0)