Daniel Pokorný

Posted on Jun 11

# I Analyzed 4,000 AI Product Recommendations And Found Almost No Relationship Between Recommendations And Store Quality

#ai #ecommerce #machinelearning #webdev

Most discussions around AI commerce focus on visibility.

Can ChatGPT find my brand? Can Gemini mention my products? Can Perplexity recommend my store?

Those questions matter.

But I became curious about a different question.

Does being recommended by AI actually have anything to do with how readable and machine friendly a store is?

To find out, I ran a small experiment.

The Setup

I used a single AI model and asked 20 high intent beauty shopping questions.

Examples included:

Best vitamin C serum
Best moisturizer for oily skin
Best anti aging skincare

Each prompt was executed 20 times.

The final dataset contained:

400 prompt runs
4,000 recommendations
238 unique brands

I then matched every recommended brand against its measured AI Commerce Score.

The Result

Across measured brands, the correlation between recommendation frequency and AI Commerce Score was:

r = 0.17

In practical terms, recommendation frequency and store quality appeared largely unrelated.

The Surprising Part

Some of the most frequently recommended brands had some of the weakest stores.

Brand	Recommendation Frequency	AI Commerce Score
Clinique	38.5%	14
SkinCeuticals	31.8%	14
Kiehl's	25.5%	14

All three stores scored only 14 out of 100.

Deep in what I classify as AI Invisible Risk.

Yet they continued to appear repeatedly in recommendations.

Meanwhile Drunk Elephant achieved the highest score in the dataset at 87 out of 100 but was not the most recommended brand.

What Seems To Be Happening

My interpretation is that today's AI systems still recommend heavily from memory.

Large brands have accumulated decades of awareness.

They appear in training data.

They appear in articles.

They appear in reviews.

They appear across the web.

As a result, recommendation systems already know them.

That knowledge appears to outweigh store quality in many recommendation scenarios.

At least for now.

Brands Beat Retailers

One finding surprised me even more.

Retailers barely appeared.

Amazon.

Sephora.

Ulta.

Combined, they accounted for only 14 recommendations out of 4,000.

AI overwhelmingly recommended brands rather than stores.

That suggests recommendation systems currently think more like category advisors than shopping directories.

Why This Matters

Today AI recommendations appear to be driven largely by familiarity.

Tomorrow may look different.

As AI systems increasingly browse websites directly, compare products, verify information, and eventually complete transactions on behalf of users, store quality may become significantly more important.

A brand can survive on recognition.

An AI agent cannot act on recognition alone.

It needs information. It needs structure. It needs trust signals. It needs something it can actually understand.

The Bigger Question

If recommendation frequency and store quality are largely disconnected today, what happens when AI systems move from remembering brands to evaluating businesses in real time?