I publish free leaderboards of which products AI answer engines name when someone asks them to recommend software in a category. The method is deliberately boring: take one buying question, write it 44 different ways, ask every engine all 44, count every product named across the answers, publish the counts and the raw runs.
Last week I posted a result that I could not explain: two boards over what I had assumed was one market — small-business CRM and open-source CRM — came back with nothing in common in their top 20s. Not reordered. Zero shared products.
The obvious objection, and the one I got, is that this says more about my engines than about the question. LLM output is noisy. Maybe I had measured two engines having a bad day.
So I ran the controls. There are two knobs — the engine and the wording — and you can hold each one still and turn the other.
The setup
| small-business CRM | open-source CRM | |
|---|---|---|
| phrasings asked | 44 | 44 |
| answers | 132 | 88 |
| engines | ChatGPT, Gemini and Perplexity | Gemini and Perplexity |
| distinct products named | 65 | 29 |
No product name appears anywhere in any question, on purpose. If you seed the question with vendors you get your own assumptions back.
Three of the small-business 44, verbatim:
What is the best CRM software for a 10-person small business?
Best CRM for a small sales team
We're a 15 person agency managing sales leads in spreadsheets and it's a mess. What CRM should we buy?
Two of the open-source 44:
What is the best open source CRM software for a small sales team?
Best self-hosted CRM for a 20-person company
Control 1 — hold the questions, change the engine
Same 44 questions, one board, 3 engines answering them independently. Rank each engine's answers on its own and compare the top tens:
| pair | products in common, top 10 |
|---|---|
| ChatGPT vs Gemini | 8 of 10 |
| ChatGPT vs Perplexity | 7 of 10 |
| Gemini vs Perplexity | 7 of 10 |
Swapping the engine leaves at least 7 of the top 10 standing every time, head included. The engines broadly agree.
Control 2 — hold the engine, change the wording
Now the other knob. One engine, asked both question sets:
| products in common | |
|---|---|
| top 10 vs top 10 | 0 of 10 |
| all 37 vs all 13 | 1 — Odoo Community |
Zero. One engine, one week, one extraction pipeline, 37 product names on one side and 13 on the other, and the two top tens do not touch. Across every product either run named, the entire intersection is Odoo Community — which is exactly the product you would predict, because it is the one sold in both shapes.
The same comparison on the full published boards, which is 132 + 88 answers rather than one engine's share of them: 0 products in common in the top 20, 2 across all 65 and 29 — Odoo (Community Edition) and Vtiger.
That is the finding. The engine is the small knob. The wording is the big one. A leaderboard of "which CRM do AI assistants recommend" is not measuring a market, it is measuring a phrasing family, and swapping families does not reorder the list — it replaces it.
The sharpest single number: SuiteCRM sits at rank 2 on the open-source board, named in 44 of 44 phrasings of that question and 81 of its 88 answers — and 0 times in 132 answers to the small-business one. Same product, same week. Counted twice, independently: the extractor's vendor rows, and a case-insensitive regex for suite-?crm straight over the raw answer text. The two agree, so there is no third number in play. On the single engine held fixed above it is 44 of 44 and 0 of 44.
Then someone told me both my question sets were wrong
I took the small-business board to the SuiteCRM community forum rather than publish a zero about somebody's project and say nothing. A consultant with twenty years of CRM projects replied, and the useful half of what he said was that my questions were not wrong in their wording, they were empty:
Additionally, your questions contain way too little context for a real business scenario.
What are your specific requirements? Do you have any particular needs in a specific
industry? What are your existing system?If you ask AI with this context and some more specific requirements (highly sensitive
customer data, can't be stored in a cloud or SaaS) you'll find way more answers containing
SuiteCRM.
That is a falsifiable claim, so I ran it. Each of the 44 board questions was rewritten as its twin, index for index: same buying ask, same register, plus an industry, an existing-systems detail and a sensitivity obligation. Question 0 became:
We are a 10-person law firm and client matter records are confidential under our professional duty, so our partners will not allow them to sit on a third party's servers; we already run our own file server and mail in the office. What is the best CRM software for us?
The rule the rewrite was done under is the one that makes the result worth anything: a question may describe a buyer's situation and may never describe a product's category. No "open source", no "self-hosted", no "on-premise", no "data sovereignty", and no vendor name from the board. Writing SuiteCRM's own positioning back at the engine would have returned 44 of 44 and measured nothing. A script greps all 44 questions for those phrases and for every one of the 65 vendor names on the board, and it ran before a single API call. It caught one: question 11 said "pipeline CRM", and Pipedrive is on the board. Rewritten, re-run, zero hits.
Engines, extraction prompt and aliases were loaded out of the board's own config at import rather than copied, so the only thing that changed was the question.
He was right, and it is not close.
| engine | SuiteCRM, no context | with context |
|---|---|---|
| ChatGPT | 0 / 44 | 42 / 44 |
| Gemini | 0 / 44 | 33 / 44 |
| Perplexity | 0 / 44 | 8 / 44 |
| all three | 0 / 132 | 83 / 132 |
It goes from absent to first on ChatGPT and joint first on Gemini. And the top of the board moves with it — top-10 overlap between the board and its context twin, per engine: 4 of 10 on ChatGPT, 4 of 10 on Gemini, 6 of 10 on Perplexity. The context run names 113 distinct products against the board's 65.
The part I did not expect, and the reason I am writing this up
Look down the "with context" column rather than across it.
The same 44 questions, the same added constraint, both runs inside the same hour — and one engine moves 42 of 44 while another moves 8. That is not a small disagreement about ranking. The engines disagree about how much a buyer's stated constraint should change the answer at all, and they disagree by more than they disagree about the answer itself.
Which inverts the tidy conclusion I had two sections ago. Wording beats engine — but how much wording beats engine is itself engine-dependent. If you are evaluating LLM output at any scale, both halves of that matter: a prompt-sensitivity result measured on one model is not a fact about models, and a model comparison run on one prompt is not a fact about the models either. The interaction term is the biggest thing on the table and it is the one nobody publishes.
What I can't claim from this
I would rather put the limits in the post than in a footnote.
- The context run is not published as a board. The two boards below are live with their raw answers; that third run is not on the site yet, so those three numbers are the only ones here you cannot go and check yourself today. I am telling you which is which rather than blurring them together.
-
Two model families and one search product, named on each board's own
ranking.json. No Claude, no Grok — no API key for either, a limit and not a choice. Two of the three model ids I requested are floating aliases; each run records the pinned id the API actually returned. - No time series, and I am not going to imply one. Each board carries a second run as a repeatability check rather than a second date — and the small-business board's two runs are not even contemporaneous, because I re-measured it when I added a third engine. So a product at 1 mention is a level, not a decline. Nothing here is rising, falling or fading.
- Named is not recommended. I count that a product appeared in an answer to a buying question. "Consider X", "X is common but", and a bare list item all count the same. Read it as share of shelf, not endorsement.
- One family each. 44 phrasings is a wide sample of one intent, not a market. There are certainly qualifiers I have not tried that would split a list again — that is rather the point.
- The rewrite is mine. I wrote the 44 context twins, under the rule above and with the grep as a check, but a different person writing them would get a different number.
The raw data
- Small-business CRM board — https://connexion.me/c/crm/?v=1443328a0bcf
- Open-source CRM board — https://connexion.me/c/oscrm/?v=1438f5015a25
Each board links its own raw files at the bottom: answers-runA.jsonl is the untouched engine responses, mentions-runA.jsonl every extraction, ranking.json the table. Click a product name and you get the questions that named it; click a question and you see the verbatim text each engine returned. Every figure above except the three context ones came out of those files.
If your project is on one of these boards and the line looks wrong to you, tell me — I would rather fix a board than defend one. That is not a figure of speech: this whole post exists because someone told me my questions were bad in public and he was right.
Disclosure, up front rather than buried: connexion.me is mine, the boards and the raw runs are free, and there is a paid monitoring subscription linked from each board page. Nothing here is behind it.
Top comments (0)