I have been running a small experiment for a few weeks, and this one genuinely stopped me for a second.
More and more buyers ask ChatGPT or Claude for a tool recommendation before they ever open Google. So I have been measuring what the models actually say when someone asks for the best tool in a category, and how much the answer moves when you change nothing but the wording.
Shared inbox software gave me the clearest example so far.
I asked two versions of the same question, 10 times each, on ChatGPT, Claude, Perplexity and Gemini. First the plain category label, "best shared inbox software". Then the way a real person actually types it, "how do we collaborate on shared email inboxes without losing our normal email workflow".
On the label question, Missive got named on all 10 runs. On the reworded one, ChatGPT named Missive zero times out of ten, while it kept naming Front and Help Scout on all ten. Same company, same site, same ten runs. The only thing that changed was the sentence.
To be fair it was not every model. Claude actually leaned the other way and named Missive more on the reworded question. But the ChatGPT drop was clean and it repeated across all ten runs.
The thing I keep taking from these is that a single check will lie to you. If I had asked once, seen Missive missing, and stopped there, I would have sworn they had some deep AI problem. They do not, they are 10 out of 10 on the other phrasing. One question on one model on one run tells you almost nothing.
And the tools that held did not do anything clever. Their pages just describe the actual job in the words a buyer uses, so the model can place them no matter how the question is asked.
I wrote up the full data, every number and every model, here: https://www.bersyn.com/blog/shared-inbox-phrasing-flip
I am running one of these teardowns roughly every week, poking at a different category. If that is your kind of rabbit hole, follow along. And I am curious, in your own category, is the recommendation this sensitive to phrasing, or did shared inbox just happen to be a swingy one?
Top comments (5)
a tool going from 10 out of 10 to zero just from rewording the question is kind of unsettling. makes me wonder how much of the score is the tool vs the phrasing. what was the wording that tanked it?
Good questions, and the second one is easy so let me give you the exact wording first.
Label version: "best shared inbox software."
Reworded version: "how do we collaborate on shared email inboxes without losing our normal email workflow."
That is the whole change. Same tool, same site, same ten runs, just the sentence.
On the bigger question, how much is the tool versus the phrasing, the honest answer is that almost none of it is the tool in that moment. The product did not get worse between the two questions. What changed is which evidence the model goes and retrieves. The label question pulls the roundups and best-of lists, and Missive is on those, so it shows up every time. The reworded one is a workflow problem, not a category, so the model reaches for pages and threads that actually discuss collaborating on a shared inbox, and Missive has less that ties it to that specific job.
So the "score" is not really measuring the tool on its own. It is measuring how well the stuff written about the tool matches the exact way the buyer asked. That is the unsettling part you picked up on, and it is also the useful part, because a 0/10 on a reword is usually a content and citation gap for that phrasing, not a verdict on the product.
And yeah, that is exactly why I run every phrasing ten times instead of once. A single pull would have told me Missive was broken, when it is 10 out of 10 one wording over.
the volatility is the real story here, and it's why AEO isn't just SEO with a new name. in search you optimize for a known query. here the "query" is whatever phrasing a buyer happens to type, which is basically infinite and you don't control it. what actually moves the model isn't on-page anything, it's how often you show up in the corpus it learned from and retrieves against: comparison posts, reviews, threads where real users name you. so the fix for a 0/10 reword usually isn't the wording, it's not being cited widely enough that the model reaches for you regardless of how the question gets asked.
Completely with you on the corpus point, that is the bigger lever for most companies and it is exactly why this is not SEO with a new coat of paint. You do not control the query, so you cannot optimize a page for it the way you would a keyword.
The one thing I would add, and it is the reason I run every phrasing multiple times, is that the data lets you tell two different failures apart. If a company is zero across every phrasing and every model, that is the corpus gap you are describing, they are simply not cited enough anywhere, and no amount of rewording their own site fixes it. But Missive was ten out of ten on one phrasing. The model clearly has it in the corpus and reaches for it easily. Its zero on the reword is not an absence of citations, it is that the citations and its own pages tie it to the category label instead of the workflow the buyer actually described.
So I think it is both, and the shape tells you which one you are dealing with. Absent everywhere is a get-cited problem, your version. Present on the label and gone on the job is a get-cited-for-the-right-job problem, where the corpus still matters but it has to be corpus about the specific workflow, not just more mentions of the category. Curious whether you see that split too, or whether you would still put almost all of it on raw citation volume.
Yes, and your split is sharper than what I wrote. Absent everywhere versus present-on-the-label-but-gone-on-the-job really are different failures, and the shape does tell you which one you have. I was collapsing both into volume.
The one I would add is a third shape, and you are the person with the data for it since you already run each phrasing several times: present on both phrasings but unstable across runs. Same wording, same model, different answer run to run. That is neither absence nor mis-association, it is sitting close to the retrieval threshold where it is nearly a coin flip. It matters because it looks like a win roughly half the times you check, so it tends to get filed as solved.
On fixing your second shape, I think the lever is different in kind rather than more of the same. Moving from the category label to the job does not need more citations, it needs citations whose surrounding language is the buyer's own description of the workflow. That pushes you away from the artifacts that look most like marketing. A directory listing or a comparison table seats you next to the category noun. A messy forum thread where someone describes their actual problem and a stranger replies with your name seats you next to the job. Fewer of the second kind tends to do more than more of the first.