Run a prompt through ChatGPT today, then run the exact same words tomorrow. You may see different brands named, different sources cited, and a different winner at the top. Nothing about your website changed. The model just rolled the dice again.
That non-determinism is the single most misunderstood thing about showing up in AI answers — and it quietly wrecks the way most marketers try to measure their presence. If you checked your brand in Gemini once last month and called it "tracked," you measured a coin flip, not a trend. Tools like Sourceable exist precisely because a one-shot look at an AI answer tells you almost nothing.
Quick answer: why do AI answers change every time?
AI answers are non-deterministic. The same engine, given the same question, can return different sources, brands, and ordering from one run to the next — and different engines barely agree with each other. One analysis of local queries found only about 35% of cited domains repeat between runs, meaning roughly two-thirds vanish the next time you ask. Across engines, an audit of 680 million citations found only ~11% of domains overlap between ChatGPT and Perplexity. Because of this variance, a single AI-visibility check is unreliable. You need continuous tracking across many runs and engines to see your real position.
The variance is bigger than a rounding error
It's tempting to assume the wobble is minor — a source swapped here, a reorder there. The data says otherwise.
When researchers compared how the major engines source information, the disagreement was structural, not cosmetic. Google AI Overviews and Google's own AI Mode — same company, same index — cited the same URLs only 13.7% of the time. Push across companies and it gets wider: on open-ended queries, ChatGPT and Google AI Overviews shared roughly 4% of their cited sources. Even when researchers pooled everything ChatGPT cited, that combined set matched only about 25% of the sources Google put front and center. Three-quarters of Google's primary sources didn't appear in ChatGPT's answer at all.
The engines also pull from different worlds. In one study, Reddit and forums made up around 25% of ChatGPT's citations, while Google AI Overviews drew only about 4% from community sources and leaned on vendor and competitor pages for roughly 45%. So "we get cited a lot" in one engine can mean "we're invisible" in another.
Brand citation rates swing just as hard. A 2026 study of 34,234 AI responses found a 46-times gap between platforms: ChatGPT named brands in just 0.59% of responses, while Perplexity did so 13.05% of the time. Same brands, same questions — wildly different odds of getting mentioned depending on where you look.
Why this breaks the "quick audit" habit
Most teams still measure AI visibility the way they measured a keyword ranking: check it once, screenshot it, move on. That worked when a #3 ranking stayed #3 for weeks. It falls apart when the "ranking" is regenerated probabilistically every time someone asks.
Here's the trap. You run your prompt, ChatGPT names your competitor, and you conclude you've lost. Or it names you, and you conclude you've won. Both conclusions could be noise. Prompt drift — the tendency for the same question to surface different brands week to week, and sometimes run to run — means a single observation has a huge margin of error baked in. Acting on it is like judging a coin as "biased toward heads" after one flip.
This is also why rolling ChatGPT, Gemini, Perplexity, and Google AI into one blended "AI visibility score" can mislead you. A single averaged number hides the only thing that's actionable: which engine you're winning, which you're losing, and why. You can be dominant in Perplexity and absent in ChatGPT and see a perfectly mediocre composite that tells you to do nothing.
What reliable measurement actually looks like
If the signal is noisy, the answer isn't to give up — it's to sample properly. A few principles hold up:
Run each prompt many times, not once. One check is an anecdote; dozens of runs start to reveal a real rate. If your brand shows up in 8 of 20 runs, that 40% presence is a number you can track and defend — far more useful than a single yes/no.
Track per engine, never blended-only. Keep ChatGPT, Gemini, Perplexity, Claude, and Google AI as separate lines. The whole point is to see where you win and where you don't.
Watch the trend, not the snapshot. Because month-to-month drift is real, direction matters more than any single day's reading. Is your presence rate climbing after you published that comparison page, or flat?
Separate movement from noise. This is why serious GEO tools now report confidence intervals rather than a bare percentage — so you can tell a genuine gain from ordinary variance before you spend budget chasing it.
This is the model Sourceable is built around: repeated sampling across ChatGPT, Claude, Gemini, and Perplexity, tracked per engine over time, so you see whether your brand's mention rate is actually moving — not whether you got lucky on one prompt.
Why bothering pays off
There's a reason this measurement discipline is worth the effort: AI traffic converts. Reported figures put ChatGPT visitors at a 15.9% conversion rate, Perplexity at 10.5%, and Claude at 5% — against roughly 1.76% for traditional organic search. Being present in the answer, reliably and in the engines your buyers actually use, is worth real revenue. But you can't improve a rate you're measuring with a single lucky look.
FAQ
Does ChatGPT give everyone the same answer?
No. Answers are non-deterministic and vary run to run, and can also shift based on phrasing, timing, and personalization. The same prompt can name different brands and cite different sources on repeat runs.
How many times should I run a prompt to trust the result?
Enough to see a stable rate rather than a single outcome — think dozens of runs per prompt per engine, not one. Continuous tracking beats periodic spot-checks because it captures drift over time.
Is a blended AI visibility score useful?
Only as a rough headline. Averaging every engine into one number hides where you're actually winning or losing. Keep per-engine breakdowns so you know which platform to fix.
Why do different AI engines cite such different sources?
They index differently and weight sources differently — ChatGPT leans on community discussion like Reddit, while Google AI Overviews favors vendor and competitor pages. Overlap between engines can be as low as ~4–11%.
The takeaway
Your brand's spot in AI answers isn't a fixed ranking you can screenshot. It's a probability that shifts every time someone asks, and shifts again depending on which engine they use. Measure it like a probability — many runs, per engine, over time — and the noise resolves into a trend you can act on. Measure it once and you're just reading tea leaves.
Want to see your real, per-engine mention rate instead of a single lucky snapshot? Track your brand across AI models with Sourceable.
Top comments (0)