Originally published on The Searchless Journal
If you have been tracking your brand's AI visibility with a dashboard that shows citation share percentages, brand mention counts, or ranking positions across ChatGPT, Perplexity, and Gemini, the numbers on your screen are probably wrong. Not because the tool is broken. Not because the data is fabricated. The numbers are wrong because they are based on too few measurements, and the system being measured is inherently probabilistic. A new paper from IQRush, set for release next week, lays out the mathematical case, and a separate research team reached the same conclusion in April. The implication for the GEO industry is significant: most AI visibility tracking currently in production is indistinguishable from noise.
The Core Problem: Generative Models Are Non-Deterministic
When you ask ChatGPT, Perplexity, or Gemini the same question twice, you frequently get different answers. Different sources cited. Different brands mentioned. Different framing. This is not a bug. Generative models are designed with temperature settings that introduce randomness into each response. The model samples from a probability distribution over possible next tokens, which means the same input produces different outputs on different runs. This is the property that makes these models feel creative and conversational. It is also the property that makes tracking visibility across them fundamentally different from tracking rankings in traditional search.
In traditional SEO, Google returned the same ten blue links for the same query roughly the same way for every user. Personalization added some variation, but the core ranking was stable. You could check your position, check it again next week, and trust that movement reflected something real. AI search does not work this way. Each query is a fresh draw from a probability distribution. The sources cited in one response are one possible set from many possible sets. A single measurement tells you what happened on one roll of the dice.
How Much These Numbers Actually Move
The IQRush paper, authored by Ron Sielinski, co-founder of IQRush, builds on his earlier work examining citation variability. In that prior study, he showed that when testing SearchGPT on running gear queries, Tom's Guide appeared in approximately 9.5 percent of citations while Runner's World accounted for roughly 6.0 percent. On a dashboard, Tom's Guide looks like the clear winner. But the margin of error on these measurements was large enough that the 3.5-point difference was not statistically meaningful. The two sites could easily be tied, or Runner's World could be ahead. You cannot tell from a single sample.
This is not an edge case. This is the default state of AI visibility measurement. SparkToro's widely cited research from earlier this year found that AI tools give different recommended brand lists more than 99 percent of the time you ask the same question. Rand Fishkin, who led that study, advised that before spending money on AI visibility tracking, you should make sure your provider shows their math. The IQRush paper provides the framework for what that math should look like.
When A Ranking Is Worth Trusting
The paper establishes two conditions that must both be true before an AI visibility ranking can be considered reliable. First, the ranking order must stabilize. When you start querying an AI engine on a given topic, the top cited sources change frequently as new responses are added. It is only after enough data accumulates that the leaders begin to separate from the pack. Second, the gap between the top sources must exceed the margin of error. If two sites have citation rates of 8 percent and 7 percent respectively, but the margin of error on each is plus or minus 3 percent, you cannot claim one outranks the other.
Both conditions must hold simultaneously. Neither alone is sufficient. In the IQRush study, across 30 platform-topic combinations, the number of cited responses needed for both conditions to be met ranged from 33 to 94. That is cited responses specifically, not total queries, because many AI responses do not include citations at all. The practical implication is straightforward: for each keyword or topic you want to track, you need to run the query dozens of times, record every cited source, and compute confidence intervals before you can say anything meaningful about your visibility relative to competitors.
Three of the thirty tests conducted in the paper never reached stability, even after 125 queries. All three were on SearchGPT, where the top sources remained too close together to separate statistically. There is no universal cutoff point. What works for tracking visibility on Perplexity for a B2B software query may not work for tracking visibility on Gemini for a consumer product query. Each platform-topic pair requires its own sample size calculation.
Why This Breaks Most Tracking Tools
The majority of AI visibility tracking tools on the market today run a query once, maybe a handful of times, and report the results as if they were fixed facts. A dashboard showing that your brand has 12 percent citation share on Perplexity for a given query is displaying a single data point from a wide distribution. The true citation share could be anywhere from 5 percent to 20 percent. Without confidence intervals and repeated measurements, the number is a snapshot, not a measurement.
This is particularly dangerous in competitive reporting. If your dashboard shows you at 12 percent and your competitor at 9 percent, it looks like you are winning. But if both numbers have margins of error of plus or minus 4 percent, the difference is not real. Acting on it, allocating budget based on it, or reporting it to a client as a competitive advantage is building on sand.
The problem compounds when you look at trends over time. A week-over-week change from 10 percent to 14 percent looks like growth. If each number is based on a single measurement, the change is likely within the noise floor. You cannot distinguish real improvement from random fluctuation without enough samples to compute meaningful confidence intervals for each time period.
What Reliable AI Visibility Tracking Looks Like
The IQRush paper provides a stopping rule. You keep querying until two things happen: the ranking order stops changing and the gaps between positions exceed the margins of error. In practice, this means building a data collection pipeline that runs each tracked query repeatedly over time, logs every cited URL, and computes cumulative citation rates with confidence intervals. Only when both stability conditions are met do you report a ranking.
This is more expensive than running each query once. It requires more API calls, more data storage, more processing. But the alternative is reporting numbers that are mathematically indistinguishable from random noise. For agencies selling AI visibility tracking as a service, the choice is between doing the work correctly or selling a dashboard that misleads clients.
Fishkin's advice applies here. Before subscribing to an AI visibility tracking tool, ask the vendor how many times they query each keyword. Ask whether they show confidence intervals. Ask what their stopping rule is. If they cannot answer these questions, the numbers they are showing you are single samples from a distribution, and any ranking derived from them is provisional at best.
The Broader Implication For GEO As A Discipline
Generative Engine Optimization as a practice depends on measurement. You cannot optimize what you cannot measure, and you cannot measure what you cannot distinguish from noise. The field is young enough that most practitioners have not yet confronted this problem head-on. The tools are new, the methodologies are borrowed from traditional SEO where rankings were stable, and the clients want clean numbers on clean dashboards.
The honest answer is that AI visibility measurement is fundamentally harder than traditional rank tracking. It requires statistical methods, sample size calculations, and an understanding of confidence intervals that most SEO professionals were never trained in. The IQRush paper is a contribution to solving this problem, but it is also an indictment of the current state of the industry. If your tracking methodology cannot distinguish signal from noise, it is not tracking. It is storytelling.
For brands investing in GEO, the takeaway is not to stop tracking. It is to track honestly. Run queries multiple times. Calculate margins of error. Demand that your tools and vendors do the same. The numbers that result will be less precise, less clean, and less impressive on a dashboard. They will also be true, which is the only foundation worth building a strategy on.
Top comments (0)