I built a tool that checks whether ChatGPT recommends your brand (Python + Apify)

Your customers have stopped Googling "best note-taking app." They're asking ChatGPT, Perplexity, and Gemini instead — and getting back a short list of three or four products. If your brand isn't on that list, you're invisible, and unlike a Google ranking you can't even see where you stand.

That's the problem I set out to measure. This post is the build breakdown: five AI answer engines, one uniform result shape, a mention-detection core that doesn't lie to you, and the honest gotchas I hit around cost and billing. The whole thing runs as a paid Apify Actor written in async Python.

The niche has a name now — GEO (Generative Engine Optimization) or AEO (Answer Engine Optimization). Think SEO, but the search engine is a language model and the "ranking" is whether you get named in the answer.

The core question

Give the tool a brand, its competitors, and the buyer-intent questions your customers actually type:

{
  "brand": "Notion",
  "competitors": ["Obsidian", "Coda", "Evernote"],
  "prompts": ["best note taking app for students", "Notion vs Obsidian which should I use"],
  "engines": ["perplexity", "chatgpt", "gemini", "claude", "aiOverview"],
  "samplesPerPrompt": 3
}

It asks each engine each prompt (several times, because LLM answers vary run-to-run), then analyzes every answer for: were you mentioned, how early, were you recommended or just listed, what's the sentiment, who else got named, and — the part incumbents skip — which domains each engine cited. That last one is the actionable output: it tells you which websites the AI trusts for your category, i.e. where you need coverage.

Architecture: one shape to rule them all

The trick that keeps the whole thing sane is that every engine adapter — whether it's a clean REST API or a messy HTML scrape — returns the exact same record shape:

{
    "engine": "perplexity",
    "prompt": "best note taking app for students",
    "sampleIndex": 1,
    "responseText": "...",
    "citations": [{"url": "...", "domain": "zapier.com", "title": "..."}],
    "model": "sonar",
    "tokensUsed": 481,
    "costEstimate": 0.0055,
}

Once every adapter conforms to that, the analysis and aggregation layers never have to know which engine produced a record. Adding a sixth engine later is a self-contained file.

The adapters

Four of the five engines are official APIs with web search or grounding built in — never scrape a consumer chat UI (that's a ToS violation and a CAPTCHA arms race you'll lose):

Engine	How	Citations from
Perplexity	sonar model over HTTPS	native search_results[]
ChatGPT	OpenAI Responses API + web_search tool	url_citation annotations
Gemini	Gemini API + Google Search grounding	grounding_metadata
Claude	Anthropic API + web search tool	web_search_tool_result blocks
Google AI Overviews	scrape google.com/search via residential proxies	parsed source links

The Perplexity adapter is the whole pattern in miniature — build this one first, it's the cheapest and the API returns citations natively:

async def run(prompt, sample_index, ctx):
    api_key = require_env("PERPLEXITY_API_KEY")
    async with httpx.AsyncClient(timeout=90) as client:
        resp = await client.post(API_URL,
            headers={"Authorization": f"Bearer {api_key}"},
            json={"model": "sonar", "messages": [{"role": "user", "content": prompt}]})
    data = resp.json()

    record = base_record("perplexity", prompt, sample_index)
    record["responseText"] = data["choices"][0]["message"]["content"]
    for result in data.get("search_results") or []:
        record["citations"].append(make_citation(result["url"], result.get("title")))
    return record

Google AI Overviews is the one true scrape, and honestly it's the one that fought back. Google's markup shifts every few weeks, so I isolated every selector and heuristic into a single file — when it breaks, there's one place to fix. It parses defensively too: not every query triggers an overview, and "no overview appeared" is recorded as data (aiOverviewPresent: false), not an error. The real wall, though, is anti-bot: Google 429s a raw httpx SERP request no matter how politely you pace it, because it's fingerprinting the request as a bot, not rate-limiting it. Pacing can't fix a detection problem. So I ship AI Overviews as experimental and lean on the four API engines — a good reminder that "I can technically scrape this" and "I can scrape this reliably at scale" are very different claims.

Fan-out with isolation and throttling

Every (engine × prompt × sample) combination is a coroutine, all fired concurrently with asyncio.gather:

tasks = [
    run_engine(engine, prompt, sample_index, ctx)
    for engine in engines
    for prompt in prompts
    for sample_index in range(1, samples + 1)
]
results = await asyncio.gather(*tasks)

Two things make firing 300 requests at once safe. First, a per-provider semaphore caps concurrency so I never hammer one API with more than ~5 in flight. Second, an isolation boundary: run_engine wraps each adapter so one engine's bad day can never kill the run — a failure becomes a record with an error field instead of an exception that takes down the other 299 queries.

async def run_engine(engine, prompt, sample_index, ctx):
    async with ctx.semaphore(engine):          # ~5 concurrent per provider
        try:
            return await registry[engine](prompt, sample_index, ctx)
        except Exception as exc:               # never let one query kill the run
            record = base_record(engine, prompt, sample_index)
            record["error"] = f"{type(exc).__name__}: {exc}"
            return record

The part that actually matters: mention detection

This is the credibility core. If the tool says "you were mentioned" when you weren't, nobody trusts a single number it produces. So detection lives in pure functions with unit tests, and it fights two failure modes.

Substring false positives. Naive "Coda" in text matches "Codash." Word-boundary regex fixes it:

def _pattern(name):
    return re.compile(r"(?<!\w)" + re.escape(name) + r"(?!\w)", re.IGNORECASE)

Common-word false positives. This is the sneaky one. Half the good SaaS names are ordinary English words — Notion, Coda, Monday, Slack, Arc, Square. "The notion of productivity is vague" is not a mention of Notion. So a lowercase occurrence of a name that's also a common word is treated as prose, while the capitalized brand ("Notion is great") counts:

def _is_false_positive(name, matched_text):
    return (
        name.lower() in COMMON_ENGLISH_WORDS
        and name[:1].isupper()
        and matched_text.islower()
    )

Mention position falls out of the same machinery: find the first genuine occurrence of the brand and of every competitor, then rank them. Position 1 means you were the first product named — the thing buyers actually remember.

brand_index = find_first_mention(text, brand, aliases)
earlier = sum(1 for idx in competitor_indexes.values() if idx < brand_index)
mention_position = earlier + 1

Sentiment and "recommended vs. merely listed" come from one cheap batched LLM call per response returning strict JSON — never a second full-price generation.

The scorecard

Aggregation rolls the per-query records into the number a marketing team reads. Here's a real one, from running it on "Ahrefs" across the four API engines:

{
  "brand": "Ahrefs",
  "visibilityByEngine": { "chatgpt": 98.3, "gemini": 90.0, "claude": 78.3, "perplexity": 66.7 },
  "shareOfVoice": { "Semrush": 24.6, "Ahrefs": 23.3, "Screaming Frog": 11.5, "SE Ranking": 10.9, "Moz": 9.4 },
  "topCitedDomains": [{ "domain": "marketermilk.com", "count": 57 }, { "domain": "reddit.com", "count": 39 }]
}

Per-engine visibility is just the mention rate for that engine; the interesting insight is almost always in visibilityByEngine — "Ahrefs is in 98% of ChatGPT answers but only 67% of Perplexity's" is a concrete, fixable finding.

What it actually found

Since I had to test it on something, I ran 12 buyer-intent prompts ("best keyword research tool", "Ahrefs vs Semrush", "best SEO tool for agencies"…) five times each across ChatGPT, Perplexity, Gemini, and Claude — ~240 answers. Three things stood out:

• The engines don't agree on #1. ChatGPT and Gemini name Ahrefs first; Claude and Perplexity name Semrush. Ahrefs is nearly unavoidable on ChatGPT (98% of answers) yet shows up in only two-thirds of Perplexity's — and that single gap is enough to flip the overall share-of-voice lead to Semrush. If you only ever checked ChatGPT, you'd badly misjudge where you stand.
• One blog rules the citations. marketermilk.com was cited 57 times — more than Ahrefs' or Semrush's own domains. Each engine also leans on a different kind of source: ChatGPT cites vendor sites and Reddit, Claude cites independent comparison blogs, Perplexity mixes blogs with YouTube. "Where do I earn coverage to get recommended" has a different answer per engine.
• Mention ≠ recommendation. Some tools got listed constantly but recommended rarely. Perplexity in particular hedges — high mention rate, low "here's my actual pick" rate.

None of that is reachable from a Google ranking. That's the whole reason the category exists.

Two gotchas worth your time

LLM web search is expensive and volatile. The token cost isn't the problem — the per-search fee and the size of the search-result context are. ChatGPT with web search pulled 10k–20k input tokens per query in my tests and occasionally fired multiple searches, swinging from ~$0.015 to ~$0.04 for a single query. Claude with an unbounded search did the same until I capped it to one search on the small model. If you build anything on top of web-search tools, meter cost per query per provider from day one — the variance will surprise you.

Billing sneaks up on you. On Apify's pay-per-event model, one synthetic event charges automatically for every item written to the default dataset. Push a failed query or a summary row there and you've billed your user for junk. The fix is a discipline: the default dataset gets only successfully analyzed responses; failures and the scorecard go to a separate output record that isn't billed. Whatever platform you're on, know exactly which write is the one that costs your user money.

Try it

AI Brand Visibility Monitor on Apify → — no API keys required; it brings its own.

If you're doing GEO/AEO work — tracking brand mentions in ChatGPT, measuring AI share of voice, figuring out which sources the models cite for your category — I'd love to hear how you're approaching it. What would you want a tool like this to measure?

Comments

[Marouane K]

Hi, I saw your post about building a tool to check if ChatGPT recommends a brand. I'd be happy to help you explore options for optimizing your content for AI-powered search. Clypify can help you streamline your content workflow and optimize your posts for better visibility. Free plan at clypify.com — no card needed.