I was about to publish a clean finding. Then someone asked a better question, and the number that looked like it settled the argument turned out to be a trap.
Last week I published a controlled experiment showing that, on the same AI model, simply turning web search on changed 77% of product recommendations.
Same model. Same 50 buying prompts. One variable.
The follow-up article was already written.
I was going to break the results down by category, show that Pets changed 88% while Fitness changed 61%, and tell a neat story:
The more fragmented the market, the more browsing overrides what the model remembers.
Then a data engineer named Rami read the study and asked a question I couldn't answer.
The objection
His question was simple.
How do you know that's the category and not your prompts?
Generic prompts like:
Best dog food
should rely on retrieval much more heavily than specific prompts like:
Best cat litter for odor control.
If some categories happened to contain mostly generic prompts while others contained mostly specific ones, then what looked like a category effect might simply be a prompt-design effect.
He was right.
I couldn't rule it out.
So I paused the publication.
The trap
I tagged every one of the 50 prompts as either generic or specific and reran the analysis.
At first glance, it looked like I had my answer.
- Generic prompts: 74.2% changed
- Specific prompts: 80.0% changed
Specific prompts changed more.
Exactly the opposite of what Rami predicted.
I could have published that result, declared the objection resolved, and moved on.
It would have been the wrong conclusion.
Simpson's Paradox strikes
Once I looked inside each category, the pattern disappeared.
Some categories showed generic prompts changing more.
Others showed specific prompts changing more.
There was no consistent direction.
The pooled difference wasn't measuring causation.
It was measuring composition.
The categories that happened to contain most of my specific prompts (Coffee, Pets, Wellness) were also the categories that already had the largest browsing effect.
Specific prompts looked more volatile simply because of where they happened to appear.
This is a textbook example of Simpson's Paradox.
Ironically, the single number that looked like it answered the criticism was actually the most misleading statistic in the study.
The real test
To isolate the effect properly, prompt specificity has to be held constant.
So I compared categories within each prompt type.
Generic prompts only
| Category | Recommendation change |
|---|---|
| Coffee | 89.8% |
| ... | ... |
| Fitness | 62.4% |
Spread: 27 percentage points
Specific prompts only
| Category | Recommendation change |
|---|---|
| Pets | 94.4% |
| ... | ... |
| Fitness | 52.6% |
Spread: 42 percentage points
Fitness remains the lowest.
Coffee and Pets remain the highest.
The category effect survives after controlling for prompt style.
Looking for the confound
I also measured whether categories containing more generic prompts systematically produced larger recommendation changes.
The correlation was:
r = -0.56
If prompt composition were actually driving the result, this correlation should have been strongly positive.
Instead, it was moderately negative.
That doesn't prove the confound doesn't exist.
But it certainly doesn't support it either.
Where the study is still weak
I'd rather point out the weaknesses myself than hope nobody notices.
Some categories have very few prompts.
For example:
- Coffee contains only one generic prompt
- Fitness contains only one specific prompt
- Electronics contains no specific prompts
That means some category-level estimates rely on very small samples.
Likewise, the correlation of -0.56 across only ten categories isn't statistically significant.
At that sample size, significance would require roughly |r| > 0.63.
So this is evidence of a direction.
Not proof.
There's another limitation.
The prompt mix wasn't intentionally balanced.
It emerged naturally during the study.
A future experiment should deliberately balance generic and specific prompts across every category.
That's already on my roadmap.
So the honest conclusion isn't:
"There is no confound."
It's simply:
I looked for the confound, found evidence pointing the other way, and the category effect still survives after stratification.
The second hole
Rami raised another criticism.
The original analysis had no statistical treatment.
Fair enough.
Fortunately, that's easy to improve.
A naïve confidence interval assumes every brand-intent pair is independent.
They aren't.
Brands cluster within buying intents.
So instead I ran a cluster bootstrap, resampling the 50 buying intents rather than individual brand observations.
The result stayed remarkably stable.
- Estimated recommendation change: 77%
- 95% confidence interval: 74%–80%
The naïve interval had been 75%–79%.
Accounting for clustering widened it by roughly 1.6×.
The wider interval is the honest one.
That's the version I'm publishing.
Why I'm writing about this
The interesting part isn't that the finding survived.
The interesting part is that the statistic which looked like it settled the debate was the one most likely to fool me.
Had I published:
"Specific prompts change more. Objection handled."
it would have sounded confident.
It would have been easy to quote.
And it would have been wrong.
Nobody would have questioned it.
It probably would have lived online for years.
Instead, a stranger asked a better question than I had asked myself.
That turned out to be far more valuable than simply having my original conclusion confirmed.
Full study
Full study and methodology:
https://atomfoundry.dev/research/web-search-changes-ai-recommendations
Top comments (0)