"What is the best CRM software for a small business?" Ask four AI tools and you'd expect four similar lists. We asked ChatGPT, Claude, Gemini and Perplexity the same "what is the best..." question in 33 categories, from CRM to password managers, and compared every name they gave.
They put the same brand first in only 12 of the 33. This post is the method: how we called each API, the prompt, how we turned free-text answers into comparable lists, and the things that broke along the way.
The results, from our full study.
The setup
We asked one question per category on October 17 to 18, 2026, through each provider's official API. We asked 60 categories in all; the study uses the 33 where all four tools answered through the API.
| Tool | API | Model | Web search |
|---|---|---|---|
| ChatGPT | OpenAI Responses API | gpt-5-mini | Off |
| Claude | Anthropic Messages API | claude-haiku-4-5 | Off (max_tokens 1200) |
| Gemini | Gemini API, generateContent | gemini-3.8-flash | Off (thinking level "low") |
| Perplexity | Perplexity API | sonar | On |
Temperature wasn't set, so each runs at its provider default. Perplexity is the only one searching the web live, which is why it's the only one with citations.
One limit worth stating up front: each tool was asked each question once. This is a snapshot, not a measure of how much answers drift between runs.
The prompt
The user message is the category question exactly as a buyer would type it. The system instruction is identical for all four:
You are answering a real customer who typed this question. Answer directly
with a ranked list of specific businesses, products or services by name,
with one line on each. Never ask a clarifying question; if the question is
ambiguous, pick the most common meaning and answer that.
That last sentence earned its place. Without it, some answers asked the user a question back instead of ranking anything, which leaves you nothing to compare.
From prose to lists
Each answer is free text, so the first job is pulling names out in order. We take the numbered or bulleted items (or the bold names when there's no list), in the order they appear, and keep up to 10 per answer.
That cap matters when you read the results. ChatGPT averaged 9.5 names, close to the ceiling, so its real lists may run longer:
Perplexity, the one searching live, named the fewest: 4 on average.
Merging names is the hard part
"YNAB" and "You Need A Budget" are the same product. On our first pass they counted as two, each named by fewer tools, which quietly breaks every agreement number. The fix is two steps: a spelling key, then a hand-checked alias list (about 40 pairs) for rebrands, short forms and plan names.
const spell = (n) => n.toLowerCase()
.replace(/\b(crm|software|app|inc|the|suite)\b/g, "")
.replace(/[^a-z0-9]+/g, "");
const ALIASES = {
"YNAB": ["You Need A Budget"],
"Kit (ConvertKit)": ["Kit", "ConvertKit"],
"Empower": ["Personal Capital"],
};
const CANON = new Map(Object.entries(ALIASES).flatMap(([name, others]) =>
[name, ...others].map((o) => [spell(o), name])));
const key = (n) => spell(CANON.get(spell(n)) ?? n);
spell makes "HubSpot CRM" and "Hubspot" collide on purpose. The alias map catches what spelling can't: renames and nicknames.
What we deliberately did not merge: sibling products. KeePassXC isn't KeePass, and Bigin isn't Zoho CRM, even though one company makes both. Merging those would overstate agreement.
Brands are then ranked by how many of the four tools named them, then by average position. Here's one category side by side:
Same question, four lists. All four named HubSpot, Zoho CRM and Pipedrive. See the CRM breakdown.
What broke
- A model that 404'd. Gemini 2.5 Flash returned 404 for new API keys, so we moved to gemini-3.8-flash.
- Hidden thinking tokens. Gemini's default thinking spent about 545 hidden tokens, billed as output, on a seven-word answer. Setting thinking to "low" fixed it.
- A retired endpoint. Perplexity's old Sonar chat/completions endpoint had been retired; we switched to its newer API.
- Clarifying questions. Covered above: the "never ask" line in the system prompt.
- Duplicate brands. "YNAB" vs "You Need A Budget", fixed by the alias list.
The whole run costs about $0.05 per category for all four answers.
What the sources say
Perplexity listed sources in 33 of its answers. Review sites and the brands' own pages dominate:
If you want to show up in a live-search AI's answer, the review sites it reads matter as much as your own page.
Every category
The full table, with links to each category's breakdown and every tool's complete list, is on the study page. Here's the top of it:
Accounting software: all four put QuickBooks Online first. AI meeting assistants: only Otter.ai made every list, and Perplexity broke ranks with Granola as its first pick.
The data is shared under CC BY 4.0, so feel free to chart it with a link back.
Run it on your own product
If you want to see where your product lands in these answers, our free AI visibility checker asks ChatGPT, Gemini, Claude and Perplexity your customers' questions and shows who they name instead. And if you're comparing tools that track this weekly, we compared the best AI visibility tools on price and coverage.
Here's what that looks like, in 90 seconds:
The short version
- Same prompt, four official APIs, one sample each; only Perplexity searched the web.
- A system prompt that forbids clarifying questions is what makes answers comparable.
- Name merging (spelling key plus aliases, siblings kept apart) is where agreement numbers are won or lost.
- The tools agreed on first place in 12 of 33 categories.
If you've run something like this, what did you do about run-to-run variation?
Top comments (0)