Why I started counting citations
At Orbator we ask ChatGPT, Claude, Gemini, Perplexity, and Grok the same buyer-style questions — "best CRM for a 10-person startup," "best CI/CD tool for a monorepo" — across 271 software categories, every week. We don't just log the answer. We log every source each engine cites before it gives that answer. That's the AI Recommendation Index, and it's been running long enough now that the numbers stopped looking like noise and started looking like a pattern.
The number that made me stop and re-check the query
Over the trailing 28 days, we captured 143,424 citations. 46.2% of them resolve to vendor-owned domains — comparison pages, "top 10" listicles, and buyer's-guide content published by companies that sell a product in the exact category being asked about. Not third-party review sites, not Reddit threads, not analyst reports. The vendors themselves, showing up as the evidence behind what looks like neutral advice.
I want to be careful here: this isn't proof of manipulation. It's proof of a structural blind spot. Vendor content is often the most detailed, most frequently updated, most SEO-optimized material on a given category — so it's exactly the content a retrieval-augmented model is most likely to find and cite. The AI isn't lying. It's citing what's easiest to find, and what's easiest to find is marketing.
The second finding that surprised me more
We also measure engine agreement — how often ChatGPT, Claude, Gemini, Perplexity, and Grok converge on the same top recommendation for a category. They fully agree only about 16% of the time. So the "AI recommendation" a buyer sees depends heavily on which chatbot they happened to open. That's not a minor caveat for anyone building a product and hoping to get recommended consistently.
What this means if you build or buy software
If you're a developer choosing a tool based on an AI's suggestion, it's worth asking what's underneath the answer, not just trusting the answer. If you're building a product, understand that your own comparison pages may already be shaping what AI tells your prospective customers — for better or worse.
All of this — per-category shares with 95% Wilson confidence intervals, engine-agreement rates, the full citation dataset — is published under CC BY 4.0.
How do you personally decide whether to trust an AI's tool recommendation — do you check the sources, or just go with it?
Top comments (1)
Built this because I kept getting AI recommendations for tools in my own space and had no idea what was actually behind them — turns out a lot of it is vendor content citing itself. Curious how the rest of you decide what to trust when an AI recommends a tool: do you actually check the sources, or just go with the answer?