When querying Large Language Models for product or brand recommendations, developers and marketers often assume top-tier models converge on a shared ground truth.
To test this assumption empirically, we conducted a benchmark across 10 major LLM architectures to evaluate entity retrieval variance, inter-model consensus, and RAG divergence in a niche market context: luxury jewelry brands in Romania for wedding and engagement rings.
Here is what the data showed.
🛠️ Experimental Methodology
To isolate model behavior and prevent prompt-induced bias, we established a strict protocol:
- Prompt: "Ce branduri de bijuterii de lux din România îmi recomanzi pentru verighete și inele de logodnă? Dă-mi un top 5, cu un argument scurt pentru fiecare și sursele pe care te bazezi." (Zero-shot, asking for Top 5, short justification, and cited sources)
-
Brand Hints:
0(No brands mentioned in the prompt). - Session State: Stateless, fresh context window, zero history/custom instructions.
- Execution: Single run per model (no cherry-picking or regeneration).
- Models Evaluated (N=10): ChatGPT (OpenAI), Claude (Anthropic), Gemini (Google), Copilot (Microsoft), Grok (xAI), Perplexity AI, DeepSeek, Kimi (Moonshot), GLM (Zhipu), and Qwen (Alibaba).
📊 Key Findings & Metrics
Across 10 models, there were 50 total available ranking slots (10 models × 5 slots).
1. High Entity Entropy
- 29 distinct entities were returned across the 50 slots.
- 19 out of 29 brands (66.2%) were recommended by only a single model.
- No single brand achieved 100% reach across all 10 models.
2. Inter-Model Agreement
- The average pairwise overlap between any two arbitrary models was 18% (an average of 0.91 shared entities per 5 recommendations).
-
Top-ranked entity:
TEILORappeared in 6 out of 10 models (Average rank: 2.0, 3× #1 placements). -
Runner-up:
Malvenskyappeared in 5 out of 10 models (Average rank: 1.2, 4× #1 placements).
3. The Copilot Anomaly (RAG Divergence)
Microsoft Copilot exhibited an overlap coefficient of 0.00 relative to the other 9 models. It returned a set of entities completely unique to its search context, highlighting how divergent web search grounding mechanisms (Bing RAG pipeline) drastically alter LLM outputs compared to base model training data or alternative web indices.
📈 Aggregated Leaderboard
| Rank | Brand | Model Frequency | Market Coverage (%) | Avg Position (Lower = Better) | #1 Placements |
|---|---|---|---|---|---|
| 1 | TEILOR | 6 / 10 | 60% | 2.0 | 3 |
| 2 | Malvensky | 5 / 10 | 50% | 1.2 | 4 |
| 3 | Sabion | 3 / 10 | 30% | 2.0 | 0 |
| 4 | Coriolan | 3 / 10 | 30% | 2.3 | 1 |
| 5 | KULTHO | 3 / 10 | 30% | 3.7 | 0 |
| 6 | Sabrini | 3 / 10 | 30% | 3.7 | 0 |
🧠 Technical Implications for AEO / GEO
For engineers building Answer Engine Optimization (AEO) or Generative Engine Optimization (GEO) pipelines, this benchmark highlights several critical systems insights:
- Deterministic Consensus is an Illusion: In niche or regional e-commerce queries, model outputs do not collapse into a single consolidated knowledge graph. Output entropy remains high.
- RAG vs. Parametric Memory: Models leaning heavily on real-time web search (Perplexity, Copilot) diverge significantly from standard base models depending on web index freshness and search query decomposition.
-
Structured Data Visibility: Entities with strong digital footprints, explicit schema markups, and widespread press mentions (
TEILOR,Malvensky) consistently bypass RAG filtering barriers across multiple model architectures.
🔗 Resources & Raw Data
You can inspect the full raw dataset (CSV) and detailed breakdowns on the study page:
- Study URL: Websem AEO Research: Luxury Jewelry Study
- Dataset: Raw response logs and screenshots archived for replication.
What techniques are you using to measure LLM visibility or RAG consistency across regional datasets? Let's discuss in the comments below!
Top comments (0)