I've spent a good part of this year building a system that watches how brands show up across ChatGPT, Claude, Perplexity, Gemini, and Google's AI Overviews. It scans all five every day, and it's improving daily.
The intent was to build a multi-tenant, multi-user, enterprise-grade system designed to track thousands of domains and millions of prompts with deep reporting and analytical capabilities, with connections to Google Search Console, Google Analytics, keyword tracking, and more.
It would have been a massive project had we not already had a multi-tenant architecture built on Google Cloud, with dozens of integrations and robust reporting systems in place.
When I started, I assumed this was a scraping side-project with a bit of prompt plumbing on top.
It was a bit bigger than that. This is a builder's first account of the terrain: which platforms give you an API, which do not, and the gap between "I can query the model" and "I can see what a real user actually gets."
I am not going to hand out the finished blueprint, partly because parts of it are our proprietary product, and partly because half of what worked six months ago is already wrong.
Disclosure: I work on PureSEM's AI visibility product, and part of what's described below runs in production there. I'll point back to it once, at the end, and otherwise this piece is about the general problem, not the product.
Table of Contents
- The naive version and the robust version that just keeps growing
- Platform by platform
- What does "visibility" mean when the answer changes every time?
- What is still unsolved
The naive version and the robust version that just keeps growing
Here is where everyone is going to start.
Hit an API, ask the question a customer would ask, and check whether the brand shows up.
(This assumes you've already built out your prompt set of course, which is a whole other problem I may cover in another article.)
from openai import OpenAI
client = OpenAI()
resp = client.chat.completions.create(
model=MODEL, # whatever's current; the specific model is the least stable part of this
messages=[{"role": "user", "content": "What's the best CRM for startups?"}],
)
answer = resp.choices[0].message.content
print("acme" in answer.lower()) # visible? true/false
You ship that, times three or four other APIs, run it across a few hundred prompts, and put a green or red dot next to each brand and store it in a table.
There are two things wrong with this.
One: that API call is not what a user sees.
The consumer ChatGPT app and the raw model API are different stacks. The app has its own retrieval, its own system prompt, and its own product surface, and it behaves differently depending on whether the person is on the free tier or a paid tier.
Free leans on static training data with a knowledge cutoff. Paid runs live web search. So a plain model call answers a question nobody actually asked: "what would the bare model say," not "what does the product show my customer." Those are not the same question, and the gap between them is the actual job.
Two: you run that script five times and you get five different answers.
Different brands are named, different ordering, and sometimes different sources. A single call told me nothing I could trust. The green dot is a coin flip.
Once both of those landed, the project stopped being a scraper and became a sampling-and-inference problem with a nasty data-collection layer underneath.
Platform by platform
Here's the state of the market for mid-2026. This will be stale by winter.
Perplexity is the friendliest to a builder. The Sonar API is OpenAI-compatible, so you barely change your client code, and it returns citations as first-class metadata instead of something you have to regex out of prose. One thing worth knowing before you build on it: Perplexity has put Sonar into maintenance mode and now points new integrations at its Agent API instead — same web-grounded behavior, plus support for third-party models. Sonar still works today, but check which one you actually want before you write code against it.
from openai import OpenAI
pplx = OpenAI(api_key=KEY, base_url="https://api.perplexity.ai")
resp = pplx.chat.completions.create(
model="sonar-pro",
messages=[{"role": "user", "content": "best CRM for startups"}],
)
# resp includes citations alongside the answer text
The catch is the same one that follows us through this whole piece. The API result is close to, but not identical to, what the Perplexity app shows a logged-in human. It's useful, but it's not ground truth.
Google Gemini has a proper API through Vertex AI and AI Studio, including grounding with Google Search. Again, it's queryable, and again it's a different thing from the Gemini consumer app — and very much a different thing from what Gemini produces when it powers Google's search surfaces.
OpenAI (ChatGPT) and Anthropic (Claude) both offer solid model APIs with web-search tools you can enable.
You can get a model plus live retrieval. What you cannot get through the API is the consumer product's exact retrieval and ranking behavior, which is the behavior that decides whether your customer sees the brand.
And the consumer surfaces move on their own schedule. As an example, on May 7, 2026, ChatGPT quietly started surfacing clickable brand links directly in answers instead of burying them in citation chips — a single product change that moved referral traffic at scale for the brands tracking it, and had nothing to do with any API you were calling.
Google AI Overviews is where it gets much more difficult and expensive.
There is no Google API that lets you ask "show me the AI Overview for this query in this location." AI Overviews are generated inside the search results page. They trigger conditionally — industry trackers put it near half of all queries, far higher in health and B2B tech, lower in ecommerce and finance — and they vary by geography, device, and signed-in state.
Google added GenAI performance data to Search Console this year, which gives you impressions but not the queries that triggered them, and only for your own property. It tells you nothing about competitors and nothing about what the block actually said.
So to observe AI Overviews you have to scrape the search results page. Either you build your own headless-browser rig or use a third-party Search Engine Results Page (SERP) API that parses the block for you.
Both are real categories, both work, and both are more fragile than anything else in this process.
As a taste of the fragility: a well-known SERP API sometimes cannot return the Overview in the first response and hands you a page token you have to redeem in a second call, and that token expires in about four minutes. Nothing here is a stable endpoint. It is an observation of a moving target.
For each of the five surfaces we're tracking, there are two distinct questions, and the industry keeps blurring the lines between them:
- What does the model say? APIs answer this well.
- What does the product show a user? Mostly, no API answers this, and this is the one that matters commercially.
Every design in this space necessarily involves decisions about how far you are willing to go to close that gap and what you are willing to approximate.
What does "visibility" mean when the answer changes every time?
This is the part I find most worth a developer's attention, and it is the part most tools paper over.
These are probabilistic systems. Ask the same question twice, and you get two draws from a distribution, not two reads of a fixed value.
A 2026 study on measuring AI search visibility (Schulte, Bleeker, and Kaufmann) found day-to-day source overlap in the 34 to 42 percent range, meaning well over half the cited sources churn between one day and the next.
Other estimates are harsher still: one industry analysis found only about 2% of citations survived across three consecutive runs of the same prompt in the same session. Wherever the exact number lands, the direction is the same. If your tool checks a prompt once and reports "you are mentioned" or "you are not," it is reporting a single draw as if it were the weather.
So "visibility" is not a rank. It is a rate. The correct output for "does ChatGPT recommend Acme for this query" is not yes or no — it is something like "in 40% of sampled runs, with this confidence interval." Which means the unit of measurement is a sampling design, not a request.
Once you accept that, the design questions get concrete:
- How many runs per prompt before the number is stable rather than noise? More than one, and enough that you can report a confidence interval instead of a point estimate. The same paper's bootstrap analysis lands on at least 7 runs per prompt per day for brand-level detection, and at least 8 when source-level coverage matters — single digits either way. Practitioner guidance elsewhere clusters in the same range. Pick your own floor, but pick one, and report the interval. Report a single run as if it were the answer, and you are publishing noise in a clean font.
- Which axes of variance do you hold fixed, and which do you sample across? Session, geography, time of day, signed-in state, model version, and even conversational mode versus search mode all move the answer. You cannot freeze all of them, so you decide which ones are part of "visibility" for your customer and which are confounds to average out.
- How do you keep the estimate honest as the platform shifts underneath you? A model version bump or a product change can move every number overnight, and you need to be able to tell "the brand's visibility changed" apart from "the surface changed." Those look identical in the raw data, but could meaningfully change representations to a customer.
None of this is exotic math. It's just a sampling of data. The data source lies to you a little differently every time you ask.
But if you skip it, everything downstream is decoration on a random number.
What is still unsolved
I want to end where DEV is at its best, which is being straight about the parts I have not cracked.
There is no ground truth. I can observe models through APIs and observe products through the page, but I cannot see what a specific real person in a real session actually got. Everything is an estimate of a hidden distribution. I can make the estimate tighter. I cannot make it certain.
Attribution is much harder to measure. A mention is not a click, and AI surfaces usually provide answers without sending anyone anywhere.
Connecting "the brand got named more often this month" to anything a business can bank is an open problem, and most of the confident-sounding attribution numbers floating around are softer than they look.
The surfaces move faster than the tooling. Every platform in this piece shipped a behavior change while I was building against it. Anything you write, including this article, is dated the day it publishes. The right instinct is to build for change detection, not for a fixed schema, because the schema will change, and you want to notice rather than silently report the old shape.
And the core gap does not close. The APIs describe the model. The customer lives in the product. Until and unless the products expose what they actually showed — and I would not hold my breath on that — the job is a careful approximation.
As mentioned up top, this is the system running behind PureSEM's AI visibility product — same sampling problem described above, running against real brand and competitor data instead of the CRM example earlier in this piece.
If you are building in this space, I would like to hear how you are handling the run-to-run variance specifically. That is the piece where I suspect there are better ideas than mine, and it is the piece that determines whether the rest means anything.
Top comments (0)