DEV Community

Anakin
Anakin

Posted on

Measuring ChatGPT answers means measuring the product, not the model API

#chatgpt #webscraping #python #seo

Your monitoring script calls the OpenAI API with the same prompt users type into ChatGPT, but the result does not match what people see on chatgpt.com. The API returns a model response. ChatGPT the product may run a web search, rewrite the query, pick sources, add citations, and vary the answer by country. If you are trying to measure whether a brand appears in ChatGPT answers, that difference matters.

The API and the product are not the same target

The OpenAI API is the right tool when you want to build with a model. It is not a reliable way to answer questions like:

  • Did ChatGPT mention our product for this category prompt?
  • Which domains did ChatGPT cite?
  • Did the answer change in the US versus the UK?
  • Did ChatGPT run a web search, or answer from model context?

Those are product-layer questions.

For example, this prompt looks simple:

best project management tool for small engineering teams
Enter fullscreen mode Exit fullscreen mode

A direct model call might return a generic list based on training data and whatever tools you explicitly configure. ChatGPT in the browser may decide to search the web, query something like:

best project management tools small engineering teams 2026 Linear Jira Asana ClickUp comparison
Enter fullscreen mode Exit fullscreen mode

Then it may cite review sites, vendor docs, or recent comparison posts. That retrieval behavior is the thing you are measuring. If you skip it, your numbers can look precise while measuring the wrong system.

What usually breaks when you scrape it yourself

The direct approach is browser automation. You open ChatGPT with Playwright, submit a prompt, wait for the streamed answer to finish, parse the DOM, and store the result.

That can work for small internal checks. It gets annoying when you put it on a schedule.

Common failure modes are not subtle:

TimeoutError: locator.waitFor: Timeout 30000ms exceeded
Enter fullscreen mode Exit fullscreen mode

or:

Error: Execution context was destroyed, most likely because of a navigation
Enter fullscreen mode Exit fullscreen mode

Sometimes the script does not crash. It just captures half an answer because the stream was still running. That is worse, because bad data enters your metrics quietly.

You also have to handle login state, anti-bot checks, rate limits, changing selectors, regional proxies, and answer variants. A selector like this may work today:

const answer = await page.locator('[data-message-author-role="assistant"]').last().innerText();
Enter fullscreen mode Exit fullscreen mode

Then ChatGPT ships a frontend change, and your job starts returning empty strings.

For this specific measurement problem, Wire exposes ChatGPT scraping as an async action that returns the answer text, citations, and whether web search ran.

Store answers as evidence, not just scores

A useful tracking pipeline should keep the raw answer and the parsed fields. Do not store only mentioned: true and throw everything else away. You will want to debug why a score changed.

A minimal record might look like this:

{
  "prompt": "best project management tool for small engineering teams",
  "country": "us",
  "run_at": "2026-08-14T10:30:00Z",
  "answer_text": "For small engineering teams, I would shortlist Linear...",
  "web_search_triggered": true,
  "web_search_query": "best project management tools small engineering teams 2026 Linear Jira Asana",
  "citations": [
    {
      "url": "https://example.com/project-management-tools",
      "domain": "example.com",
      "title": "Best Project Management Tools for Engineering Teams",
      "position": 1,
      "cited": true
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

That gives you three useful measurements:

  1. Mention share: how often the answer names your brand.
  2. Source share: how often ChatGPT cites your domain.
  3. Competitor/source analysis: which brands and publishers show up when you do not.

Here is a small Python pass over stored responses:

import json
from pathlib import Path

BRAND = "linear"
DOMAIN = "linear.app"

runs = [json.loads(p.read_text()) for p in Path("runs").glob("*.json")]

def mentions_brand(run):
    return BRAND.lower() in run["answer_text"].lower()

def cites_domain(run):
    return any(
        DOMAIN in citation.get("url", "")
        for citation in run.get("citations", [])
        if citation.get("cited")
    )

total = len(runs)
brand_mentions = sum(mentions_brand(run) for run in runs)
domain_citations = sum(cites_domain(run) for run in runs)

print({
    "total_runs": total,
    "mention_share": brand_mentions / total if total else 0,
    "citation_share": domain_citations / total if total else 0,
})
Enter fullscreen mode Exit fullscreen mode

This is intentionally boring. The hard part is not the aggregation. The hard part is collecting comparable answers reliably over time.

Treat location as part of the input

ChatGPT answers can change by market. If your customers are in Canada, Germany, and Australia, one US-based run does not tell you enough.

Make country a first-class field in your prompt schedule:

{
  "prompt": "best payroll software for remote teams",
  "countries": ["us", "ca", "gb", "au"],
  "frequency": "weekly"
}
Enter fullscreen mode Exit fullscreen mode

Then report by country instead of averaging everything together too early. Otherwise a strong US presence can hide that you never appear in UK answers.

If you already handle scheduling and storage, Wire can sit in the collection layer where each prompt and country pair becomes an async job with structured output.

Be careful with the metric

A ChatGPT visibility score is not a source of truth. It is a sampled measurement of a changing product.

You should expect variance from:

  • prompt wording
  • country
  • time of day
  • whether search runs
  • recently published pages
  • changes in ChatGPT retrieval behavior

The fix is not to pretend the data is deterministic. Run enough prompts, keep the raw evidence, and compare changes over consistent prompt sets.

A practical starting point: choose 25 real customer questions, run them weekly in your main markets, store answer_text, citations, web_search_triggered, and web_search_query, then review both the aggregate numbers and the raw answers before you draw conclusions.

Top comments (1)

Collapse
 
magia_cc_0f5823d6cb56ed5 profile image
magia c'c •

Keeping answer_text beside the visibility score makes those changes auditable. One fix for cites_domain: compare the parsed hostname with linear.app or its subdomains; a substring check also accepts a different host with linear.app in its path or query. I would store the resolved citation URL as well as the displayed one before calculating source share.