How much does web search change what an AI recommends?
I measured it using a fixed set of e-commerce buying questions.
The result surprised me.
Why I Ran This Experiment
A few weeks ago I published research showing something unexpected.
Across thousands of AI recommendations, store quality explained almost nothing about why brands were recommended.
Brand popularity explained much more.
Yet almost 73% of recommendation behavior remained unexplained.
After publishing the study, data engineer Rami suggested an interesting hypothesis.
Maybe part of that "unexplained" behavior isn't mysterious at all.
Maybe it's simply the retrieval layer.
His suggestion was simple:
Run the exact same prompts twice.
- Once with web search enabled.
- Once with web search disabled.
Then compare the recommendations.
So I did.
Experiment Setup
Model:
GPT-4o
Conditions:
• Browsing ON
• Browsing OFF
Dataset:
50 buying prompts
Categories:
Pets
Beauty
Supplements
Coffee
Fashion
Fitness
Food
Home
Wellness
Electronics
Runs:
10 per prompt per condition
Metric:
Unique recommended brands
To isolate the effect of browsing from the effect of model size, I also repeated the experiment using GPT-4o-mini.
The Result
The overlap between GPT-4o with browsing enabled and GPT-4o without browsing was only:
23%
That means:
77% of the recommended brands changed simply by enabling web search.
Same model.
Same prompts.
Same methodology.
One toggle.
What Changed?
With browsing disabled, GPT-4o recommends mostly from its internal memory.
Those recommendations tend to favor well-known brands the model likely encountered frequently during training.
With browsing enabled, the model relies heavily on live retrieval.
Many memory-based recommendations disappear and are replaced by brands discovered during search.
These weren't small ranking adjustments.
They were largely different recommendation lists.
Was It Really Search?
A reasonable objection would be:
Maybe GPT-4o simply recommends different brands than GPT-4o-mini.
So I isolated that variable.
Model difference (Browsing OFF)
Moving from GPT-4o-mini to GPT-4o changed only about:
6% of recommendations
Search difference (Same model)
Turning browsing ON changed:
77% of recommendations
The retrieval layer had a dramatically larger impact than model size.
It Depends on the Category
Browsing didn't affect every category equally.
| Category | Recommendations Changed |
|---|---|
| Pets | 88% |
| Fitness | 61% |
The pattern appears consistent.
Markets dominated by a handful of famous brands showed smaller changes.
Markets with many niche brands showed much larger changes.
Pets has consistently produced the strongest effects across every study I've run.
I still don't fully understand why.
The Bigger Black Box
Rami's original hypothesis had another part.
He suggested that much of the unexplained variance may live inside the model's training history.
Questions like:
- Which sources mentioned a brand?
- How often?
- In what context?
- With what sentiment?
Those signals aren't observable today.
I can measure what gets recommended.
I cannot yet measure why a model internally trusts one brand more than another.
That remains one of the biggest black boxes in AI recommendation systems.
Why This Matters
Many companies treat "AI visibility" as a single objective.
I don't think it is.
There are at least two different systems at work.
Memory
Long-term exposure during training.
This rewards brands that are already widely known.
Retrieval
Live information gathered at query time.
This rewards brands that are discoverable and well represented across the web.
Those are different mechanisms.
And they often produce different recommendations.
Key Takeaway
If your optimization strategy only targets one of these layers, you're missing the other.
Depending on whether an AI assistant uses browsing, retrieval may completely replace memory-driven recommendations.
In this experiment, enabling browsing changed 77% of the recommended brands.
That's too large to ignore.
Methodology
- 50 fixed ecommerce buying prompts
- 10 executions per prompt
- GPT-4o with browsing ON
- GPT-4o with browsing OFF
- GPT-4o-mini comparison
- Recommendation overlap measured using distinct recommended brands per buying intent
Browsing implementations evolve over time, so treat the percentages as a snapshot rather than a universal constant.
The important finding isn't the exact number.
It's the magnitude of the effect.
I'm continuing this research through Atom Foundry, where I study how AI systems understand, evaluate, and recommend ecommerce brands.
If you spot weaknesses in the methodology or have ideas for improving the experiments, I'd genuinely like to hear them.
Top comments (0)