I publish leaderboards that measure which products AI assistants name when someone asks them to recommend software in a category. The method is deliberately boring: take one buying question, write it 44 different ways, ask each engine all 44, count every product named across the answers, publish the counts and the raw runs.
Last week I ran that method twice over what I had assumed was one market. The two vendor lists came back with nothing in common — not "reordered", not "one newcomer", zero products shared between the two top 20s.
The only difference between the two runs was two words in the question.
What was asked
Two boards, same two engines, same shape.
| small-business CRM | open-source CRM | |
|---|---|---|
| phrasings asked | 44 | 44 |
| answers | 88 | 88 |
| engines | Gemini, Perplexity | Gemini, Perplexity |
| distinct products named | 53 | 29 |
No product name appears anywhere in the questions, on purpose — if you seed the question with vendors you get your own assumptions back. The list has to be whatever the engine says.
Three of board one's 44, verbatim:
What is the best CRM software for a 10-person small business?
Best CRM for a small sales team
We're a 15 person agency managing sales leads in spreadsheets and it's a mess. What CRM should we buy?
Three of board two's 44:
What is the best open source CRM software for a small sales team?
Best self-hosted CRM for a 20-person company
We want to host our own CRM on-premise to control customer data privacy. What open source options should we evaluate?
Same buyer, same problem, same two engines, same week. One set of questions says "open source" and "self-hosted"; the other does not.
What came back
Small-business CRM — top 10 of 53
| # | product | answers naming it | share of 88 |
|---|---|---|---|
| 1 | HubSpot CRM | 79 | 89.8% |
| 2 | Pipedrive | 75 | 85.2% |
| 3 | Zoho CRM | 48 | 54.5% |
| 4 | Less Annoying CRM | 22 | 25.0% |
| 5 | Freshsales | 14 | 15.9% |
| 6 | Salesforce | 11 | 12.5% |
| 7 | monday CRM | 10 | 11.4% |
| 8 | Salesflare | 10 | 11.4% |
| 9 | Nutshell | 8 | 9.1% |
| 10 | Close | 7 | 8.0% |
Open-source CRM — top 10 of 29
| # | product | answers naming it | share of 88 |
|---|---|---|---|
| 1 | EspoCRM | 82 | 93.2% |
| 2 | SuiteCRM | 81 | 92.0% |
| 3 | Odoo (Community Edition) | 43 | 48.9% |
| 4 | Twenty | 41 | 46.6% |
| 5 | SugarCRM | 16 | 18.2% |
| 6 | Vtiger | 16 | 18.2% |
| 7 | ERPNext | 11 | 12.5% |
| 8 | Krayin | 9 | 10.2% |
| 9 | Dolibarr | 4 | 4.5% |
| 10 | YetiForce | 4 | 4.5% |
The overlap
| comparison | products in common |
|---|---|
| top 20 vs top 20 | 0 |
| all 53 vs all 29 | 1 |
That single product is Vtiger — and it is exactly the one you would predict, because it is the one sold in both shapes: an open-source self-hosted edition and a cloud product under the same brand. On the open-source board it is 6th with 16 mentions. On the small-business board it is 52nd of 53, named in one answer out of 88.
The comparison is done on a lenient normalisation (lowercase, strip non-alphanumerics, drop parentheticals and a trailing "crm"), so the 0 is not an artefact of "HubSpot CRM" not matching "HubSpot". I went looking for false zeros and didn't find one.
The number that actually stopped me
It isn't on either table.
Across 88 answers and 53 distinct products, the small-business question returned exactly one product with an open-source edition, named exactly once. Not one open-source product in the top 20. Not one in the top 40. Vtiger, at rank 52.
Meanwhile SuiteCRM — absent from that board entirely — is named in 44 of 44 phrasings of the open-source question, and EspoCRM in 43 of 44. These are not obscure projects that the engines had never heard of. They were sitting right there, and the generic question did not reach them.
What I think this means
Not that either board is wrong. HubSpot really does appear to be what these two engines answer when a small business asks for a CRM, and that is a real fact about a real question a real buyer types.
What's wrong is the category framing — mine included. I had been treating "CRM" as the unit of measurement. It isn't. The unit is the phrasing family. Two words of qualifier didn't reorder a list; they swapped out the entire list, because they moved the buyer from one purchase to a different purchase that happens to share a noun.
If you maintain an open-source product, the practical version is this: "are we mentioned by AI assistants" is not one question with one answer. Your absence from the generic phrasing may be telling you nothing at all about you, and quite a lot about which buyer that phrasing describes. Before you read a zero as a verdict, find out which question produced it.
And if you're building anything that measures LLM output at scale, the transferable bit is that the phrasing set is the experiment. Mine is 44 phrasings, published, so you can look at exactly which buyer I asked about and decide whether that's your buyer.
What I can't claim from this
I'd rather put the limits in the post than in a footnote.
-
Two engines. Gemini and Perplexity — the two that answered these two boards. Not ChatGPT, not Claude, not Grok. Each board's
ranking.jsonrecords which engines actually answered it, and I don't name one that didn't. - No direction. Each board was measured twice, minutes apart, as a repeatability check — not on two different dates. So nothing here is rising, falling, gaining or fading. A product at 1 mention is a level, not a decline. I have no time series and won't pretend to.
- Named is not recommended. I count that a product appeared in an answer to a buying question. Some of those mentions are "consider X"; some are "X is a common choice but"; some are a list item. Counted the same. Read it as share of shelf, not endorsement.
- One family each. 44 phrasings is a wide sample of one intent, not the whole market. There are certainly qualifiers I didn't try that would split the list again.
The raw data
Both boards, both runs, every answer verbatim:
- Open-source CRM board — https://connexion.me/c/oscrm/?v=582c11c84669
- Small-business CRM board — https://connexion.me/c/crm/?v=e14dfd19c148
Each board links its own raw files at the bottom — answers-runA.jsonl and answers-runB.jsonl are the untouched engine responses, mentions-runA.jsonl is every extraction, and ranking.json is the table above. Every figure in this post came out of those files. Click a product name on the board and you get the questions that named it; click a question and you see the verbatim text each engine returned.
If your project is one of the 29 and the line looks wrong to me, tell me — I would rather fix a board than defend one. And if you know a qualifier that ought to split a market and doesn't, I'll measure it and post what comes back either way.
Disclosure, since it belongs up front rather than buried: connexion.me is mine, the boards and both raw runs are free, and there is a paid monitoring subscription linked from each board page. Nothing in this post is behind it.
Top comments (0)