DEV Community: Lana Plouffe

I pre-registered a prediction that my own finding would fail on this market. The product list held; the advice did not.

Lana Plouffe — Fri, 07 Aug 2026 09:27:41 +0000

This week I published a result I liked: adding a buyer's actual situation to a software buying question changes which products an AI engine names. It reproduced on a second market. Two markets, two hits, no misses.

That is exactly the shape of a finding that is really an artefact of the person writing the questions — which was me. So before running anything else, I picked the market where my own explanation says the effect should fail, wrote the prediction and the numbers that would count as a miss into a file, and committed it before the first API call.

This is that run. Part of the prediction held and part of it failed, and the part that failed is the more interesting half.

The claim being tested

The explanation I had been giving was specific: buyer context does not mainly reorder the incumbents, it admits a class the generic question cannot reach — the self-hosted line. On CRM and on call center software, that is what happened. Both markets have a large, mature line of software you install on your own equipment, and both times the context questions pulled that block onto the list.

A specific explanation predicts its own absence. If that is the mechanism, then a market with no such class behind its incumbents should barely move.

Amazon advertising software is that market. Every bid change in the category runs through Amazon's own advertising API, inside Amazon's cloud, executed by a company Amazon approved as a partner. There is no product you can run on your own hardware, because the thing being automated is not yours to host. I could not name a single self-hosted product in the category to track — and that inability is the prediction, not a gap in the preparation.

What I committed before running it

Read out of the file, which was committed before the first API call:

PRIMARY — SELFHOST regex, raw answer text, per run.
    predict  <= 15/132 on both runs
    MISS     >= 60/132 on either run   (halfway to the ~125 the other two markets hit)

SECONDARY — roster size vs out/amazonppc/runA (58 vendors).
    board's own run-to-run variation: 58 -> 66, +14%
    crm +68%, callcenter +69%
    predict  < +30%
    MISS     >= +50%

TERTIARY — top-10 overlap on the vendor FOLD KEY vs out/amazonppc/runA.
    board's own noise floor (runA vs runB): 9/10
    crm 4/10, callcenter 6/10 and 7/10
    predict  >= 8/10
    MISS     <= 6/10
    Declared in advance: 7/10 is a grey zone on this endpoint and will be reported as one.
    The PRIMARY endpoint decides the headline.

INHOUSE regex: MISS if >= 60/132 on either run.

The primary instrument is a regex over the raw answer text, so it does not depend on what my extractor chooses to count. Its value comes from being the same instrument on all three markets, unchanged.

Control row first

A comparison between two different question sets means nothing until you know what two runs of the same questions do. So, 44 questions, ChatGPT, Gemini and Perplexity, 132 answers per run:

control	top-10 overlap
the board vs its own second run	9 of 10
the context run vs its own second run	9 of 10

That is the noise floor. Anything near 10 of 10 is a repeat; a drop below it is the question.

Result 1: the product list did not move. The prediction held here.

market	products named, board → with context	top-10 overlap
CRM	65 → 109	4 of 10
call center	75 → 127	6 of 10
Amazon advertising	58 → 60	9 of 10

Roster growth of 3.4%, against 68% and 69% on the two markets where the effect fired. The second run gives 65 products, 12.1%, and 10 of 10. 37 of the board's 58 products are on both lists.

Per engine, never averaged — top-10 overlap between the board and its context twin:

engine	top-10 overlap
ChatGPT	8 of 10
Gemini	7 of 10
Perplexity	9 of 10

On the two markets where the class exists, the same comparison was 4 of 10 and 6 of 10. Here it is indistinguishable from asking the same questions twice.

Result 2: the answers moved a lot. The prediction failed here.

Same runs. Instead of counting which products were named, count how many answers talk about running it yourself. Same regex, all three markets:

market	board	with context
CRM	0 of 132	123 of 132
call center	9 of 132	127 of 132
Amazon advertising	0 of 132	49 of 132

I predicted at most 15. It came back 49 and 46.

The second instrument, which tracks the one escape hatch this market does have — being told to do it yourself against Amazon's own console and API — is a clean miss against the number I wrote down:

	board run A	board run B	context run A	context run B
answers mentioning the do-it-yourself route	22	21	102	105
— of which ChatGPT	12	8	41	42
— of which Gemini	4	5	41	42
— of which Perplexity	6	8	20	21

My miss line was 60. It came in at 102 and 105 against a board baseline of 22 and 21.

So what actually happened

The engines were given a buyer whose contract says no outside company may hold their trading data. In a market with a self-hosted line, they answer by naming self-hosted products. In this market there are none to name — and they did not invent any. The list of products stayed put, 9 of the top 10 unchanged.

What changed is the advice around the list. The engines started telling the buyer to run it themselves anyway: use Amazon's own console, work the bulk sheets, build against the advertising API. That is not a product recommendation, so it barely touches the ranking, and it is not nothing either — it is the engine telling a buyer in this category not to buy anything.

The 23 products on the context list that are not on the board are mostly that same answer wearing a product's clothes. The largest newcomers:

product	answers naming it
Amazon Ads Console	9
Amazon Ads Campaign Manager	7
Kenshoo	4
Marin	4

The only "class" this market had available to admit was Amazon itself.

Scoring myself against what I wrote down

Being honest about this is the entire reason for writing the prediction down first.

Roster growth — held. Predicted under 30%, got 3.4%. Miss line 50%.
Top-10 overlap — held. Predicted 8 or better, got 9 and 10. Miss line 6.
Do-it-yourself language — missed, and by the number I wrote down first, not one I chose afterwards. Miss line 60, got 102.
Self-hosting language — my prediction was wrong and my miss line was not reached. I said at most 15 and set the miss at 60; it came in at 49 and 46. I did not declare a verdict for the band in between, which is a flaw in how I wrote the pre-registration rather than a result. I am reporting it as undeclared instead of picking whichever side flatters me.

The mechanism claim I was making is too strong and the corrected version is narrower: buyer context reliably changes what the engine says about how to buy, and only changes which products it names when the market contains a class of product that the generic question was filtering out. Those are two different effects. I had been reporting them as one, because on the first two markets I looked at they happened to move together.

What I can't claim from this

This context run is not published as a board, and I am not promoting it to one on the strength of a run designed to attack my own finding. The boards linked below are live with their raw answers; the Amazon context run is not, so those figures are the ones here you cannot go and check for yourself today. I would rather say which is which than blur them.
Three engines, named on each board's own ranking.json. No Claude, no Grok — no API key for either, a limit and not a choice.
Named is not recommended. I count that a product appeared in an answer to a buying question. "Consider X" and a bare list item count the same. Share of shelf, not endorsement.
The rewrite is mine. I wrote the 44 context twins under a rule — describe the buyer's situation, never a product's features — with a grep enforcing that no question contains a vendor name from the board or any of the words the test is about. A different person writing them would get different numbers.
One of the newcomers is named by the board's own question. Question 39 on the live board already says "Amazon Campaign Manager", and the context twin inherits that wording verbatim, so it appears identically in both arms and the board's own second run names it 4 times. It is in the table because it is in the data, not because the context introduced it.
One market, chosen by me, for a reason I stated in advance. Picking the market where your effect should fail is better than not picking one, and it is still one market.
No time series. Each board carries a second run as a repeatability check, not a second date. A product at 1 mention is a level, not a decline.

The raw data

Amazon PPC software board — https://connexion.me/c/amazonppc/?v=38a45f42cb6c
Call center software board — https://connexion.me/c/callcenter/?v=f98d41d85020
The CRM context board, where the effect did fire — https://connexion.me/c/crmctx/?v=5b0cb256c95b

Each board links its own raw files at the bottom: answers-runA.jsonl is the untouched engine responses, mentions-runA.jsonl every extraction, ranking.json the table. Click a product name and you get the questions that named it; click a question and you see the verbatim text each engine returned.

If your product is on one of these boards and the line looks wrong to you, tell me — I would rather fix a board than defend one.

Disclosure, up front rather than buried: connexion.me is mine, the boards and the raw runs are free, and there is a paid monitoring subscription linked from each board page. Nothing here is behind it.

Buyer context moved 6 of the top 10 results on one market. I ran it again on an unrelated market and it moved 4.

Lana Plouffe — Fri, 07 Aug 2026 05:01:23 +0000

My last post ended on a result I liked far too much.

I run free leaderboards of which products AI answer engines name when you ask them to recommend software in a category. Take one buying question, write it 44 different ways, ask every engine all 44, count every product named, publish the counts and the raw runs.

A consultant on the SuiteCRM forum told me my questions were empty — no industry, no existing systems, no constraints — and that if I added the buyer's real situation, SuiteCRM would start coming back. I ran it and he was right by a mile: from 0 mentions in 132 answers to 83, and from absent to first in the category.

That is a great story and it is n = 1. One market, and it is the exact market the person who predicted it works in. So the only honest next move was to try to break it on a market that has nothing to do with CRM, and publish whichever way it fell.

The replication

The second market is call center / contact center software, a board I already had: 44 phrasings, 132 answers, ChatGPT, Gemini and Perplexity, 75 distinct products named.

Same procedure as the CRM run, three parts:

Each of the 44 board questions rewritten as its twin, index for index — same buying ask, same register, plus an industry, an existing-systems detail, and an obligation about where the call audio is allowed to live (a professional duty, a contract, a regulator, an underwriter's rule).
The extraction prompt, the noise regex and the aliases loaded out of the board's own config at import rather than copied, so the only variable is the question.
The tracked product named before the first API call. On the CRM run I already knew which product the prediction was about. Here I picked one off the board and wrote it down first: VICIdial, which the call center board names 1 time in 132 answers, rank 68 of 75. Whatever number came back for it is the number in this post.

Board question 0, and its twin:

What is the best call center software for a 50-person customer support team?

What is the best call center software for a 50-person customer support team at a hospital network, where every call is a patient conversation covered by the business associate agreements we sign and our compliance office will not allow the recordings to sit with an outside service provider? We run our own phone switch and servers in the building today.

The rule the rewrite is done under is the one that makes the result worth anything: a question may describe a buyer's situation and may never describe a product's category. No "open source", no "self-hosted", no "on-premise", no "sovereignty", and no vendor name from the board. A script greps all 44 questions for those phrases and for every one of the 75 product names on the board, and it runs before a single API call. On the CRM run it caught one and forced a rewrite. On this one it came back clean first time.

I also left the awkward words in. Where the board question says "cloud" or "CCaaS", the twin still says it, even though it sits oddly next to "cannot leave our network" — real buyers ask exactly that, and stripping it would have been the edit that helps me.

Result 1 — the tracked product

engine	VICIdial, board	with context, run A	run B
ChatGPT	0 / 44	11 / 44	18 / 44
Gemini	1 / 44	6 / 44	7 / 44
Perplexity	0 / 44	0 / 44	1 / 44
all three	1 / 132	17 / 132	26 / 132

Rank in the category: 68 of 75 on the board, 14 of 127 with context, 9 on the second context run. Spread across 13 and 19 of the 44 questions respectively, so it is not one question carrying the result.

Counted twice, independently: the extractor's product rows, and a case-insensitive regex for vici-?dial straight over the raw answer text. 1, 17 and 26. They agree, so there is no third number in play. The board's own second run names it 0 times.

So the effect reproduces on a market with nothing to do with CRM. Same manipulation, different category, a product picked in advance: it goes from noise to the top 15 of 127.

Result 2 — and it reproduces smaller

Here is the part that would have been easy to leave out.

comparison	products in common, top 10
call center board, run A vs run B — same questions, same engines	10 of 10
call center board vs its context twin, run A	6 of 10
call center board vs its context twin, run B	7 of 10
small-business CRM board vs its context twin	4 of 10

The first row is the control that makes the rest mean anything: ask the same 44 questions twice and the top ten comes back identical, 10 of 10. So the movement below it is the question, not the noise.

But adding the buyer's situation moved 6 of the top 10 on CRM and 4 of 10 here. And the head held: Five9 is first on the call center board and still first with context, with Talkdesk coming up from 3 to 2. On CRM the leader was displaced outright — HubSpot CRM runs that board and finishes 4th once the buyer says who they are, behind SuiteCRM, which the board never named at all.

What actually changed, then

Not the ranking. The roster.

The context run names 127 distinct products against the board's 75, and only 39 appear on both — 88 products get named that the plain question never surfaced once. The top of that list is a coherent class rather than a scatter:

named with buyer context	mentions	on the plain board
3CX	21	0
Asterisk	14	0
FreePBX	11	0
QueueMetrics	10	0
FreeSWITCH	4	0
Mitel MiContact Center	4	0

That is the self-hosted telephony stack, arriving as a block. The on-premise product lines from the big incumbents show up with context too — Cisco's Unified Contact Center Express at 20, Genesys Engage, Avaya IP Office — and I have deliberately kept them out of that table, because those vendors do have board rows under other product names and I did not want to publish a number that turns on where I drew the fold. The six above have no row on the board under any spelling.

The CRM run did the same thing in the same proportion: 113 products with context against 65 on the board, 25 in common.

That reframes the finding, and I think this version is more useful than the one I published last time. Adding the buyer's situation does not mainly reorder the incumbents. It opens the category to a class of products the generic question cannot reach — and whether that class then takes the top of the table depends on the market. In CRM it did. In call center software it got into the room and stopped there.

The engine asymmetry showed up again

Look down the columns of the first table, not across.

ChatGPT moves the tracked product to 11 of 44. Perplexity moves it to 0.

Last time I argued from the CRM run that the engines disagree about how much a buyer's stated constraint should change the answer at all, and that they disagree about that more than they disagree about the answer. I am repeating it only because it survived the replication: same ordering, same search-grounded engine at the bottom, on an unrelated market. If you are evaluating LLM output at any scale, that interaction is the term to budget for.

What I can't claim from this

The call center context run is not published as a board. The call center board itself is live below with its raw answers, and so is the CRM pair. This third run is not on the site, so its numbers are the ones here you cannot go and check yourself today. I would rather tell you which is which than blur them together.
n = 2. Two markets is a replication, not a law. It tells me the CRM result was not a fluke of one category; it does not tell me the size of the effect anywhere else.
The rewrite is mine. I wrote the 44 context twins under the rule above with the grep as a check, but a different person writing them would get a different number. That is the weakest joint in this whole method and I do not have a fix for it.
The extraction pipeline has opinions. The call center board's prompt excludes general office phone systems and PBXs, which is right for a board about contact centers and does mean some of what the engines offered a constrained buyer was counted out. It was applied identically to both runs, which is what the comparison needs.
Named is not recommended. I count that a product appeared in an answer to a buying question. "Consider X", "X is common but", and a bare list item all count the same. Read it as share of shelf, not endorsement.
Two model families and one search product, named on each board's own ranking.json. No Claude, no Grok — no API key for either, a limit and not a choice. Two of the three model ids I requested are floating aliases; each run records the pinned id the API actually returned.
No time series. Each board carries a second run as a repeatability check, not a second date. A product at 1 mention is a level, not a decline.

The raw data

Call center software board — https://connexion.me/c/callcenter/?v=3aaae02ec4f3
Small-business CRM board — https://connexion.me/c/crm/?v=6cde596c1bb4
The CRM board asked with buyer context — https://connexion.me/c/crmctx/?v=f0bf685d5113

One of the 44 twins, so you can judge the rewrite rather than take my word for it:

What call center software do mid-market companies in regulated industries actually use when their call recordings are not allowed to leave their own network?

If your product is on one of these boards and the line looks wrong to you, tell me — I would rather fix a board than defend one. The previous post in this series exists because someone told me in public that my questions were bad and he was right; this one exists because his fix worked so well that I did not trust it.

Changing the AI engine moved 3 of 10 results. Changing the question moved 10 of 10.

Lana Plouffe — Fri, 07 Aug 2026 02:08:00 +0000

I publish free leaderboards of which products AI answer engines name when someone asks them to recommend software in a category. The method is deliberately boring: take one buying question, write it 44 different ways, ask every engine all 44, count every product named across the answers, publish the counts and the raw runs.

Last week I posted a result that I could not explain: two boards over what I had assumed was one market — small-business CRM and open-source CRM — came back with nothing in common in their top 20s. Not reordered. Zero shared products.

The obvious objection, and the one I got, is that this says more about my engines than about the question. LLM output is noisy. Maybe I had measured two engines having a bad day.

So I ran the controls. There are two knobs — the engine and the wording — and you can hold each one still and turn the other.

The setup

	small-business CRM	open-source CRM
phrasings asked	44	44
answers	132	88
engines	ChatGPT, Gemini and Perplexity	Gemini and Perplexity
distinct products named	65	29

No product name appears anywhere in any question, on purpose. If you seed the question with vendors you get your own assumptions back.

Three of the small-business 44, verbatim:

What is the best CRM software for a 10-person small business?
Best CRM for a small sales team
We're a 15 person agency managing sales leads in spreadsheets and it's a mess. What CRM should we buy?

Two of the open-source 44:

What is the best open source CRM software for a small sales team?
Best self-hosted CRM for a 20-person company

Control 1 — hold the questions, change the engine

Same 44 questions, one board, 3 engines answering them independently. Rank each engine's answers on its own and compare the top tens:

pair	products in common, top 10
ChatGPT vs Gemini	8 of 10
ChatGPT vs Perplexity	7 of 10
Gemini vs Perplexity	7 of 10

Swapping the engine leaves at least 7 of the top 10 standing every time, head included. The engines broadly agree.

Control 2 — hold the engine, change the wording

Now the other knob. One engine, asked both question sets:

	products in common
top 10 vs top 10	0 of 10
all 37 vs all 13	1 — Odoo Community

Zero. One engine, one week, one extraction pipeline, 37 product names on one side and 13 on the other, and the two top tens do not touch. Across every product either run named, the entire intersection is Odoo Community — which is exactly the product you would predict, because it is the one sold in both shapes.

The same comparison on the full published boards, which is 132 + 88 answers rather than one engine's share of them: 0 products in common in the top 20, 2 across all 65 and 29 — Odoo (Community Edition) and Vtiger.

That is the finding. The engine is the small knob. The wording is the big one. A leaderboard of "which CRM do AI assistants recommend" is not measuring a market, it is measuring a phrasing family, and swapping families does not reorder the list — it replaces it.

The sharpest single number: SuiteCRM sits at rank 2 on the open-source board, named in 44 of 44 phrasings of that question and 81 of its 88 answers — and 0 times in 132 answers to the small-business one. Same product, same week. Counted twice, independently: the extractor's vendor rows, and a case-insensitive regex for suite-?crm straight over the raw answer text. The two agree, so there is no third number in play. On the single engine held fixed above it is 44 of 44 and 0 of 44.

Then someone told me both my question sets were wrong

I took the small-business board to the SuiteCRM community forum rather than publish a zero about somebody's project and say nothing. A consultant with twenty years of CRM projects replied, and the useful half of what he said was that my questions were not wrong in their wording, they were empty:

Additionally, your questions contain way too little context for a real business scenario.
What are your specific requirements? Do you have any particular needs in a specific
industry? What are your existing system?

If you ask AI with this context and some more specific requirements (highly sensitive
customer data, can't be stored in a cloud or SaaS) you'll find way more answers containing
SuiteCRM.

That is a falsifiable claim, so I ran it. Each of the 44 board questions was rewritten as its twin, index for index: same buying ask, same register, plus an industry, an existing-systems detail and a sensitivity obligation. Question 0 became:

We are a 10-person law firm and client matter records are confidential under our professional duty, so our partners will not allow them to sit on a third party's servers; we already run our own file server and mail in the office. What is the best CRM software for us?

The rule the rewrite was done under is the one that makes the result worth anything: a question may describe a buyer's situation and may never describe a product's category. No "open source", no "self-hosted", no "on-premise", no "data sovereignty", and no vendor name from the board. Writing SuiteCRM's own positioning back at the engine would have returned 44 of 44 and measured nothing. A script greps all 44 questions for those phrases and for every one of the 65 vendor names on the board, and it ran before a single API call. It caught one: question 11 said "pipeline CRM", and Pipedrive is on the board. Rewritten, re-run, zero hits.

Engines, extraction prompt and aliases were loaded out of the board's own config at import rather than copied, so the only thing that changed was the question.

He was right, and it is not close.

engine	SuiteCRM, no context	with context
ChatGPT	0 / 44	42 / 44
Gemini	0 / 44	33 / 44
Perplexity	0 / 44	8 / 44
all three	0 / 132	83 / 132

It goes from absent to first on ChatGPT and joint first on Gemini. And the top of the board moves with it — top-10 overlap between the board and its context twin, per engine: 4 of 10 on ChatGPT, 4 of 10 on Gemini, 6 of 10 on Perplexity. The context run names 109 distinct products against the board's 65.

The part I did not expect, and the reason I am writing this up

Look down the "with context" column rather than across it.

The same 44 questions, the same added constraint, both runs inside the same hour — and one engine moves 42 of 44 while another moves 8. That is not a small disagreement about ranking. The engines disagree about how much a buyer's stated constraint should change the answer at all, and they disagree by more than they disagree about the answer itself.

Which inverts the tidy conclusion I had two sections ago. Wording beats engine — but how much wording beats engine is itself engine-dependent. If you are evaluating LLM output at any scale, both halves of that matter: a prompt-sensitivity result measured on one model is not a fact about models, and a model comparison run on one prompt is not a fact about the models either. The interaction term is the biggest thing on the table and it is the one nobody publishes.

What I can't claim from this

I would rather put the limits in the post than in a footnote.

The context run was unpublished when I wrote this. It is a live board now, so this caveat is spent. When this post went up, the two boards below were live with their raw answers and that third run was not on the site, which made those three context numbers the only ones here you could not go and check yourself. That is no longer true: the run is published, with the same raw answer files as the others, at https://connexion.me/c/crmctx/?v=734ec29edd6e — so every figure in this post is now checkable against a file. I am correcting the caveat in place rather than deleting it, because it was accurate when you first read it.
Two model families and one search product, named on each board's own ranking.json. No Claude, no Grok — no API key for either, a limit and not a choice. Two of the three model ids I requested are floating aliases; each run records the pinned id the API actually returned.
No time series, and I am not going to imply one. Each board carries a second run as a repeatability check rather than a second date — and the small-business board's two runs are not even contemporaneous, because I re-measured it when I added a third engine. So a product at 1 mention is a level, not a decline. Nothing here is rising, falling or fading.
Named is not recommended. I count that a product appeared in an answer to a buying question. "Consider X", "X is common but", and a bare list item all count the same. Read it as share of shelf, not endorsement.
One family each. 44 phrasings is a wide sample of one intent, not a market. There are certainly qualifiers I have not tried that would split a list again — that is rather the point.
The rewrite is mine. I wrote the 44 context twins, under the rule above and with the grep as a check, but a different person writing them would get a different number.

The raw data

Small-business CRM board — https://connexion.me/c/crm/?v=1443328a0bcf
Open-source CRM board — https://connexion.me/c/oscrm/?v=1438f5015a25

If your project is on one of these boards and the line looks wrong to you, tell me — I would rather fix a board than defend one. That is not a figure of speech: this whole post exists because someone told me my questions were bad in public and he was right.

Two words changed 100% of the answers: what AI engines recommend for "CRM" vs "open source CRM"

Lana Plouffe — Thu, 06 Aug 2026 11:34:22 +0000

I publish leaderboards that measure which products AI assistants name when someone asks them to recommend software in a category. The method is deliberately boring: take one buying question, write it 44 different ways, ask each engine all 44, count every product named across the answers, publish the counts and the raw runs.

Last week I ran that method twice over what I had assumed was one market. The two vendor lists came back with nothing in common — not "reordered", not "one newcomer", zero products shared between the two top 20s.

The only difference between the two runs was two words in the question.

What was asked

Two boards, same two engines, same shape.

	small-business CRM	open-source CRM
phrasings asked	44	44
answers	88	88
engines	Gemini, Perplexity	Gemini, Perplexity
distinct products named	53	29

No product name appears anywhere in the questions, on purpose — if you seed the question with vendors you get your own assumptions back. The list has to be whatever the engine says.

Three of board one's 44, verbatim:

What is the best CRM software for a 10-person small business?
Best CRM for a small sales team
We're a 15 person agency managing sales leads in spreadsheets and it's a mess. What CRM should we buy?

Three of board two's 44:

What is the best open source CRM software for a small sales team?
Best self-hosted CRM for a 20-person company
We want to host our own CRM on-premise to control customer data privacy. What open source options should we evaluate?

Same buyer, same problem, same two engines, same week. One set of questions says "open source" and "self-hosted"; the other does not.

What came back

Small-business CRM — top 10 of 53

#	product	answers naming it	share of 88
1	HubSpot CRM	79	89.8%
2	Pipedrive	75	85.2%
3	Zoho CRM	48	54.5%
4	Less Annoying CRM	22	25.0%
5	Freshsales	14	15.9%
6	Salesforce	11	12.5%
7	monday CRM	10	11.4%
8	Salesflare	10	11.4%
9	Nutshell	8	9.1%
10	Close	7	8.0%

Open-source CRM — top 10 of 29

#	product	answers naming it	share of 88
1	EspoCRM	82	93.2%
2	SuiteCRM	81	92.0%
3	Odoo (Community Edition)	43	48.9%
4	Twenty	41	46.6%
5	SugarCRM	16	18.2%
6	Vtiger	16	18.2%
7	ERPNext	11	12.5%
8	Krayin	9	10.2%
9	Dolibarr	4	4.5%
10	YetiForce	4	4.5%

The overlap

comparison	products in common
top 20 vs top 20	0
all 53 vs all 29	1

That single product is Vtiger — and it is exactly the one you would predict, because it is the one sold in both shapes: an open-source self-hosted edition and a cloud product under the same brand. On the open-source board it is 6th with 16 mentions. On the small-business board it is 52nd of 53, named in one answer out of 88.

The comparison is done on a lenient normalisation (lowercase, strip non-alphanumerics, drop parentheticals and a trailing "crm"), so the 0 is not an artefact of "HubSpot CRM" not matching "HubSpot". I went looking for false zeros and didn't find one.

The number that actually stopped me

It isn't on either table.

Across 88 answers and 53 distinct products, the small-business question returned exactly one product with an open-source edition, named exactly once. Not one open-source product in the top 20. Not one in the top 40. Vtiger, at rank 52.

Meanwhile SuiteCRM — absent from that board entirely — is named in 44 of 44 phrasings of the open-source question, and EspoCRM in 43 of 44. These are not obscure projects that the engines had never heard of. They were sitting right there, and the generic question did not reach them.

What I think this means

Not that either board is wrong. HubSpot really does appear to be what these two engines answer when a small business asks for a CRM, and that is a real fact about a real question a real buyer types.

What's wrong is the category framing — mine included. I had been treating "CRM" as the unit of measurement. It isn't. The unit is the phrasing family. Two words of qualifier didn't reorder a list; they swapped out the entire list, because they moved the buyer from one purchase to a different purchase that happens to share a noun.

If you maintain an open-source product, the practical version is this: "are we mentioned by AI assistants" is not one question with one answer. Your absence from the generic phrasing may be telling you nothing at all about you, and quite a lot about which buyer that phrasing describes. Before you read a zero as a verdict, find out which question produced it.

And if you're building anything that measures LLM output at scale, the transferable bit is that the phrasing set is the experiment. Mine is 44 phrasings, published, so you can look at exactly which buyer I asked about and decide whether that's your buyer.

What I can't claim from this

I'd rather put the limits in the post than in a footnote.

Two engines. Gemini and Perplexity — the two that answered these two boards. Not ChatGPT, not Claude, not Grok. Each board's ranking.json records which engines actually answered it, and I don't name one that didn't.
No direction. Each board was measured twice, minutes apart, as a repeatability check — not on two different dates. So nothing here is rising, falling, gaining or fading. A product at 1 mention is a level, not a decline. I have no time series and won't pretend to.
Named is not recommended. I count that a product appeared in an answer to a buying question. Some of those mentions are "consider X"; some are "X is a common choice but"; some are a list item. Counted the same. Read it as share of shelf, not endorsement.
One family each. 44 phrasings is a wide sample of one intent, not the whole market. There are certainly qualifiers I didn't try that would split the list again.

The raw data

Both boards, both runs, every answer verbatim:

Open-source CRM board — https://connexion.me/c/oscrm/?v=582c11c84669
Small-business CRM board — https://connexion.me/c/crm/?v=e14dfd19c148

Each board links its own raw files at the bottom — answers-runA.jsonl and answers-runB.jsonl are the untouched engine responses, mentions-runA.jsonl is every extraction, and ranking.json is the table above. Every figure in this post came out of those files. Click a product name on the board and you get the questions that named it; click a question and you see the verbatim text each engine returned.

If your project is one of the 29 and the line looks wrong to me, tell me — I would rather fix a board than defend one. And if you know a qualifier that ought to split a market and doesn't, I'll measure it and post what comes back either way.

Disclosure, since it belongs up front rather than buried: connexion.me is mine, the boards and both raw runs are free, and there is a paid monitoring subscription linked from each board page. Nothing in this post is behind it.