DEV Community

Chaz Eden
Chaz Eden

Posted on • Originally published at apify.com

I built a tool that checks whether ChatGPT recommends your brand (Python + Apify)

#ai #python #seo #webscraping

Your customers have stopped Googling "best note-taking app." They're asking ChatGPT, Perplexity, and Gemini instead — and getting back a short list of three or four products. If your brand isn't on that list, you're invisible, and unlike a Google ranking you can't even see where you stand.

That's the problem I set out to measure. This post is the build breakdown: five AI answer engines, one uniform result shape, a mention-detection core that doesn't lie to you, and the honest gotchas I hit around cost and billing. The whole thing runs as a paid Apify Actor written in async Python.

The niche has a name now — GEO (Generative Engine Optimization) or AEO (Answer Engine Optimization). Think SEO, but the search engine is a language model and the "ranking" is whether you get named in the answer.

The core question

Give the tool a brand, its competitors, and the buyer-intent questions your customers actually type:

{
  "brand": "Notion",
  "competitors": ["Obsidian", "Coda", "Evernote"],
  "prompts": ["best note taking app for students", "Notion vs Obsidian which should I use"],
  "engines": ["perplexity", "chatgpt", "gemini", "claude", "aiOverview"],
  "samplesPerPrompt": 3
}
Enter fullscreen mode Exit fullscreen mode

It asks each engine each prompt (several times, because LLM answers vary run-to-run), then analyzes every answer for: were you mentioned, how early, were you recommended or just listed, what's the sentiment, who else got named, and — the part incumbents skip — which domains each engine cited. That last one is the actionable output: it tells you which websites the AI trusts for your category, i.e. where you need coverage.

Architecture: one shape to rule them all

The trick that keeps the whole thing sane is that every engine adapter — whether it's a clean REST API or a messy HTML scrape — returns the exact same record shape:

{
    "engine": "perplexity",
    "prompt": "best note taking app for students",
    "sampleIndex": 1,
    "responseText": "...",
    "citations": [{"url": "...", "domain": "zapier.com", "title": "..."}],
    "model": "sonar",
    "tokensUsed": 481,
    "costEstimate": 0.0055,
}
Enter fullscreen mode Exit fullscreen mode

Once every adapter conforms to that, the analysis and aggregation layers never have to know which engine produced a record. Adding a sixth engine later is a self-contained file.

The adapters

Four of the five engines are official APIs with web search or grounding built in — never scrape a consumer chat UI (that's a ToS violation and a CAPTCHA arms race you'll lose):

Engine How Citations from
Perplexity sonar model over HTTPS native search_results[]
ChatGPT OpenAI Responses API + web_search tool url_citation annotations
Gemini Gemini API + Google Search grounding grounding_metadata
Claude Anthropic API + web search tool web_search_tool_result blocks
Google AI Overviews scrape google.com/search via residential proxies parsed source links

The Perplexity adapter is the whole pattern in miniature — build this one first, it's the cheapest and the API returns citations natively:

async def run(prompt, sample_index, ctx):
    api_key = require_env("PERPLEXITY_API_KEY")
    async with httpx.AsyncClient(timeout=90) as client:
        resp = await client.post(API_URL,
            headers={"Authorization": f"Bearer {api_key}"},
            json={"model": "sonar", "messages": [{"role": "user", "content": prompt}]})
    data = resp.json()

    record = base_record("perplexity", prompt, sample_index)
    record["responseText"] = data["choices"][0]["message"]["content"]
    for result in data.get("search_results") or []:
        record["citations"].append(make_citation(result["url"], result.get("title")))
    return record
Enter fullscreen mode Exit fullscreen mode

Google AI Overviews is the one true scrape. Google's markup shifts every few weeks, so I isolated every selector and heuristic into a single file — when it breaks, there's exactly one place to fix. It also parses defensively: not every query triggers an AI Overview, and "no overview appeared" is recorded as data (aiOverviewPresent: false), not thrown as an error. That absence is something users genuinely want to know.

Fan-out with isolation and throttling

Every (engine × prompt × sample) combination is a coroutine, all fired concurrently with asyncio.gather:

tasks = [
    run_engine(engine, prompt, sample_index, ctx)
    for engine in engines
    for prompt in prompts
    for sample_index in range(1, samples + 1)
]
results = await asyncio.gather(*tasks)
Enter fullscreen mode Exit fullscreen mode

Two things make firing 300 requests at once safe. First, a per-provider semaphore caps concurrency so I never hammer one API with more than ~5 in flight. Second, an isolation boundary: run_engine wraps each adapter so one engine's bad day can never kill the run — a failure becomes a record with an error field instead of an exception that takes down the other 299 queries.

async def run_engine(engine, prompt, sample_index, ctx):
    async with ctx.semaphore(engine):          # ~5 concurrent per provider
        try:
            return await registry[engine](prompt, sample_index, ctx)
        except Exception as exc:               # never let one query kill the run
            record = base_record(engine, prompt, sample_index)
            record["error"] = f"{type(exc).__name__}: {exc}"
            return record
Enter fullscreen mode Exit fullscreen mode

The part that actually matters: mention detection

This is the credibility core. If the tool says "you were mentioned" when you weren't, nobody trusts a single number it produces. So detection lives in pure functions with unit tests, and it fights two failure modes.

Substring false positives. Naive "Coda" in text matches "Codash." Word-boundary regex fixes it:

def _pattern(name):
    return re.compile(r"(?<!\w)" + re.escape(name) + r"(?!\w)", re.IGNORECASE)
Enter fullscreen mode Exit fullscreen mode

Common-word false positives. This is the sneaky one. Half the good SaaS names are ordinary English words — Notion, Coda, Monday, Slack, Arc, Square. "The notion of productivity is vague" is not a mention of Notion. So a lowercase occurrence of a name that's also a common word is treated as prose, while the capitalized brand ("Notion is great") counts:

def _is_false_positive(name, matched_text):
    return (
        name.lower() in COMMON_ENGLISH_WORDS
        and name[:1].isupper()
        and matched_text.islower()
    )
Enter fullscreen mode Exit fullscreen mode

Mention position falls out of the same machinery: find the first genuine occurrence of the brand and of every competitor, then rank them. Position 1 means you were the first product named — the thing buyers actually remember.

brand_index = find_first_mention(text, brand, aliases)
earlier = sum(1 for idx in competitor_indexes.values() if idx < brand_index)
mention_position = earlier + 1
Enter fullscreen mode Exit fullscreen mode

Sentiment and "recommended vs. merely listed" come from one cheap batched LLM call per response returning strict JSON — never a second full-price generation.

The scorecard

Aggregation rolls the per-query records into the number a marketing team reads:

{
  "brand": "Notion",
  "visibilityScore": 87.5,
  "visibilityByEngine": { "perplexity": 100.0, "chatgpt": 50.0, "gemini": 100.0, "claude": 100.0 },
  "shareOfVoice": { "Notion": 38.0, "Obsidian": 41.0, "Evernote": 21.0 },
  "topCitedDomains": [{ "domain": "zapier.com", "count": 6 }, { "domain": "reddit.com", "count": 4 }]
}
Enter fullscreen mode Exit fullscreen mode

Per-engine visibility is just the mention rate for that engine; the blended visibilityScore is the mean across engines. The interesting insight is almost always in visibilityByEngine — "you're at 100% everywhere except ChatGPT, where you're at 50%" is a concrete, fixable finding.

Two gotchas worth your time

LLM web search is expensive and volatile. The token cost isn't the problem — the per-search fee and the size of the search-result context are. ChatGPT with web search pulled 10k–20k input tokens per query in my tests and occasionally fired multiple searches, swinging from ~$0.015 to ~$0.04 for a single query. Claude with an unbounded search did the same until I capped it to one search on the small model. If you build anything on top of web-search tools, meter cost per query per provider from day one — the variance will surprise you.

Billing sneaks up on you. On Apify's pay-per-event model, one synthetic event charges automatically for every item written to the default dataset. Push a failed query or a summary row there and you've billed your user for junk. The fix is a discipline: the default dataset gets only successfully analyzed responses; failures and the scorecard go to a separate output record that isn't billed. Whatever platform you're on, know exactly which write is the one that costs your user money.

Try it

If you're doing GEO/AEO work — tracking brand mentions in ChatGPT, measuring AI share of voice, figuring out which sources the models cite for your category — I'd love to hear how you're approaching it. What would you want a tool like this to measure?

Top comments (0)