To verify that an AI visibility tracker reports the answers real users see, run a sample of your tracked prompts in the ChatGPT UI on the same day, from the same country and in the same logged-out state. Then compare the brands and the cited domains with what the platform reported for those prompts. The check is necessary because the ChatGPT app and the model API give different answers to the same prompt. In a Surfer study of 1,000 prompts, collected on 4 August 2026, the API answer and the app answer to the same prompt shared only 21.3% to 31.6% of the brands they named, depending on the product.
Key takeaways
- A tracker that collects an engine's answers from the model API reports a different answer from the one users see. On ChatGPT, Surfer counted 12.1 cited sources per app answer and 3.1 per API answer.
- Compare the tracker with the UI, and also the UI with itself. ChatGPT answers change from day to day: in our monitoring, two answers to the same prompt one day apart shared 31% of their cited domains on average. The UI's own overlap is the baseline for the tracker's.
- 50 prompts with 3 runs each gives 150 answers on each side. That sample shows a large gap. It does not show a gap of a few points.
The check in six steps
- Pick 20 to 50 prompts that you track. Write down the country, the US state if you use one, and the day.
- Ask the vendor which surface each engine's data comes from (the app UI or the model API) and which session state it uses.
- Capture the UI answers for the same prompts, country and day, logged out, with 2 or 3 runs for each prompt.
- Export the tracker's answers for the same prompts and day: the answer text, the brands it counted and the cited URLs.
- For each prompt, calculate the overlap of brands and of cited domains: tracker against UI run 1, and UI run 1 against UI run 2.
- Compare the two overlaps. Then compare your brand's mention rate in the tracker with its rate in the UI answers.
What can capture the UI answers for the check?
cloro, which publishes this post, is one option. It is an API that returns the answers users see in ChatGPT, Gemini, Google AI Overview, Google AI Mode and Bing AI Mode. It sends each prompt through the product's UI in a logged-out session, from the country you select, and returns the answer text and the cited sources as JSON.
There are three ways to capture the UI answers:
- By hand. Use a private browser window that is not signed in, from a connection in the target country. This works for 20 prompts. At 150 answers it is slow, and you must copy every cited URL by hand.
- A capture API. You send the prompt and the country, and you get the answer and its sources as JSON.
- Your own browser automation. Playwright with a residential proxy in the target country. An earlier post on the ChatGPT UI answers lists the parts you must build.
If you use a capture API, check its success rate. A failed capture makes your sample smaller. We publish a daily benchmark that sends the same prompts to cloro and to five other providers, one attempt each, and counts the requests that returned an answer. Over 30 days, 20 October to 18 November 2026, with more than 4,600 prompts per engine and provider:
| Engine | cloro | DataForSEO | Oxylabs | SerpApi | SearchApi (11 days) | Bright Data |
|---|---|---|---|---|---|---|
| ChatGPT | 97.3% | 99.8% | 98.9% | not measured | 100% | 73.3% |
| Google AI Mode | 99.7% | 95.6% | 98.2% | 99.8% | 98.4% | 76.6% |
On ChatGPT, three of the four other providers returned an answer more often than cloro in this window. SearchApi has data for 11 days only (8 to 18 November), and on those prompts cloro returned 99.4%. The benchmark counts answers that came back. It does not check that an answer matches the UI.
Why does the API answer differ from the UI answer?
The model API returns what the model writes with the instructions and tools that you send. The ChatGPT app adds its own steps before the answer gets to the screen. Surfer lists them as "retrieval, ranking, safety filtering, personalization, and answer-shortening". The app also selects the model for each request, and it decides when to search the web.
Surfer's ChatGPT figures, from 1,000 prompts with one sample per prompt (4 August 2026):
- Words per answer: 863 through the API, 421 in the app.
- Brands per answer: 13.8 through the API, 7.9 in the app.
- Cited sources per answer: 12.1 in the app, 3.1 through the API.
- Cited domains that the API answer and the app answer shared: 4.8% in the study's text and 12.1% in its chart. At page level the study gives 4.8% to 6.2%.
The two figures for domain overlap disagree in the source, so quote the range. Both say the same thing: most domains that the app cites are not in the API answer.
A tracker that collects from the UI can still differ from what you see. OpenAI serves some logged-out ChatGPT traffic in a mobile web format with no model name and a shorter source list (documented in a guide to GEO metrics). A tracker that does not label those answers reports fewer sources for them. Country and session state change the answer too. A signed-in account can personalize answers. A logged-out session cannot.
How do you run the check?
Use prompts that you already track and that matter to your reports. Match the conditions on both sides:
| Condition | How to match it |
|---|---|
| Prompt text | Exact text from the tracker, with no edits |
| Country (and US state) | The location the tracker uses for that prompt |
| Session | Logged out, unless the vendor says it collects signed in |
| Day | The same calendar day as the tracker's run |
| Engine | ChatGPT first. Then repeat for Google AI Mode or AI Overview if you report them |
For each answer, keep two lists: the brands that the answer names and the URLs that it cites. Put brand names in one form before you compare ("HubSpot CRM" and "HubSpot" are one brand). Surfer's brand overlap went from 15.5% to 23.8% on raw names to 21.3% to 31.6% after this step, so the step changes the result.
This script calculates the two overlaps for one prompt. It uses Jaccard overlap: the items that both sets share, divided by all items in the two sets.
from urllib.parse import urlparse
def domains(urls):
"""Cited URLs to a set of bare domains."""
out = set()
for u in urls:
host = urlparse(u).hostname or ""
out.add(host.removeprefix("www."))
return out
def overlap(a, b):
"""Jaccard overlap of two sets: 1.0 is identical, 0.0 is nothing shared."""
a, b = set(a), set(b)
return len(a & b) / len(a | b) if a | b else 1.0
def compare(prompt, tracker, ui_run1, ui_run2):
"""Each answer is a dict with 'brands' (list) and 'sources' (list of URLs)."""
return {
"prompt": prompt,
"brands_tracker_vs_ui": overlap(tracker["brands"], ui_run1["brands"]),
"brands_ui_vs_ui": overlap(ui_run1["brands"], ui_run2["brands"]),
"domains_tracker_vs_ui": overlap(domains(tracker["sources"]), domains(ui_run1["sources"])),
"domains_ui_vs_ui": overlap(domains(ui_run1["sources"]), domains(ui_run2["sources"])),
}
Run it for every prompt, then take the mean of each column. Also count the share of answers that name your brand, on each side.
How many prompts and runs do you need?
One answer proves nothing, because the UI does not give the same answer twice. In our own monitoring, 193 prompts ran once a day in ChatGPT and in Google AI Mode from 10 to 17 November 2026. Two ChatGPT answers to the same prompt, one day apart, shared on average 30.8% of their cited domains (Jaccard, median 27.9%). For AI Mode the mean was 34.1%, and 27.5% of the day pairs shared no domain. Surfer's API and app answers shared 4.8% to 12.1% of their ChatGPT domains, which is well below that baseline. The two studies used different prompts and dates, so compare the levels, not the exact numbers.
So the useful number is a rate over many answers. With 50 prompts and 3 runs (150 answers), a mention rate near 30% has a 95% margin of about 7 points. Runs of the same prompt are not independent, so the real margin is wider. With 20 prompts and 3 runs (60 answers), the margin is about 12 points. A tracker that reports 30% when the UI gives 28% passes that test. A tracker that reports 30% when the UI gives 10% does not.
How do you read the result?
- Tracker overlap near the UI's own overlap. The tracker reports the answers users see, within the normal change between runs.
- Tracker overlap much lower on brands and on domains. The tracker and the UI answer from different conditions. Ask the vendor which surface, country and session state it used for that engine. Many platforms document this.
- Same brands, fewer sources. Ask how the vendor detects answers in a reduced format, and whether it labels them.
- Different mention rate for your brand. This number goes into your reports, so take the gap to the vendor with your prompt list and your captures.
Do the check again when the vendor changes its collection method, and when you add an engine.
Method and sources
This post is published by cloro, one of the capture APIs in the benchmark above. AI visibility and SEO platforms are among its customers, Surfer included, and Surfer's study credits cloro with data collection and analysis support. cloro measured every benchmark figure, its own included.
- Surfer, "Why You Cannot Track AI Visibility Through an API", last updated 4 October 2026. 1,000 prompts, five products, data collected on 4 August 2026, one sample per prompt. Surfer is a cloro customer, and the study credits cloro with data collection and analysis support.
- The provider benchmark is our own measurement, published and updated daily on a page about LLM scraping reliability. The figures above are the 30 days to 18 November 2026.
- The day-to-day overlap is our own measurement: 193 prompts about SERP APIs, LLM scraping, AI tracking and web search APIs, one answer per prompt, engine and day, captured from the product UI, 10 to 17 November 2026. Overlap is the Jaccard index of the cited domains of two answers to the same prompt on consecutive days.
- The margins use the normal approximation for a proportion, 1.96 × √(p(1−p)/n), at p = 0.3.
Top comments (0)