How I measure brand visibility across 5 LLMs without lying to myself (the engineering)

MUHAMMET İLBAŞ — Tue, 16 Jun 2026 12:28:11 +0000

"Does ChatGPT recommend my product?" sounds like a yes/no question. It is not. It's a statistics question wearing a yes/no costume, and treating it like a boolean is how every naive brand-tracking script quietly lies to the person who wrote it.

I spent a few months building a system that asks ChatGPT, Claude, Gemini, Perplexity, and Grok the same category questions on a schedule and reports whether a brand shows up. This post is the engineering — the four places where the naive version is wrong, and what I did instead. No marketing. Just the parts that were genuinely hard to get right.

Problem 1: the output is a distribution, not a value

Here's the first thing that breaks people. Ask the same model the same brand question twice:

> What are the best tools for team note-taking?
run 1 → [Notion, Obsidian, Evernote]
run 2 → [Notion, Coda, Obsidian, Microsoft OneNote]

Same prompt. Same model. Different answer. Not a bug — sampling, plus live retrieval variance on web-connected modes. Which means a single call gives you one draw from a distribution, and reporting it as "we rank #2!" is like measuring a coin by flipping it once and declaring it 100% heads.

So the unit of measurement can't be one response. It has to be a mention rate over N runs:

def visibility(brand, question, model, n=10):
    hits = 0
    for _ in range(n):
        brands = ask_and_extract(question, model)   # list of brands, ranked
        if entity_match(brand, brands):
            hits += 1
    return hits / n   # 0.0 .. 1.0

How big does n need to be? You're estimating a proportion p, so the standard error is sqrt(p*(1-p)/n). At n=10 your error bar is roughly ±15 points — fine for "are we at 0% or 60%," useless for "did we go from 52% to 58%." I settled on running enough samples to stabilize the trend and then smoothing across days with a moving average, because day-to-day jitter will otherwise generate fake "wins" and "losses" that send you chasing noise. The score you report should move slower than the underlying randomness.

Takeaway: if your brand tracker returns a number from a single API call, it's not measuring visibility. It's sampling noise and rounding it to a story.

Problem 2: if your prompt names the brand, you've already cheated

My first prompts looked like this:

How good is {{brand}} for team note-taking?

The visibility numbers were fantastic. They were also meaningless. By naming the brand, I primed the model to talk about it — I was measuring "will the model discuss a thing I explicitly handed it" (almost always yes), not "does the model bring this brand up on its own" (the only thing that matters).

The fix is brand-blind prompts. Ask the category question the way a stranger with no knowledge of your product would, and check the unprimed answer:

# wrong — primed
"Is {{brand}} a good note-taking app?"

# right — brand-blind
"What are the best note-taking apps for a small team in 2026?"

This sounds obvious written down. In practice almost every "track your AI mentions" script I've looked at smuggles the brand into the prompt somewhere, and the resulting dashboards are self-congratulatory fiction. If you build one rule into your own version, make it this one: the brand name never appears in the question.

Problem 3: "did it mention us" is an entity-resolution problem, not a substring search

The tempting implementation:

mentioned = brand.lower() in response.lower()   # please don't

Every one of these fails it:

Aliases: Zene, zene.ai, tryzene, Zene app — one entity, many surface forms.
Substring collisions: a brand named Coda matches decoder, codable, and the musical term coda. Short brand names are a minefield.
Negation / framing: "I wouldn't recommend X" contains the brand but is the opposite of a win. And being listed first is wildly different from a grudging parenthetical at the end.

So I stopped parsing free text and instead forced structured output — make the model return the ranked entities directly:

SCHEMA = {
  "type": "object",
  "properties": {
    "brands": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name":     {"type": "string"},
          "rank":     {"type": "integer"},
          "sentiment":{"type": "string", "enum": ["positive","neutral","negative"]}
        },
        "required": ["name", "rank"]
      }
    }
  },
  "required": ["brands"]
}

…then run an alias table + normalized matching on name, not on the raw paragraph:

ALIASES = {"zene": {"zene", "zene.ai", "tryzene", "zene app"}}

def entity_match(brand, extracted):
    keys = ALIASES.get(brand.lower(), {brand.lower()})
    return any(norm(b["name"]) in keys for b in extracted)

Now you also get rank and sentiment for free, which turns "are we mentioned" into "where, and how favorably." Budget real time for this layer — it's the quiet difference between a tracker that's right and one that feels right.

Problem 4: five engines, on a schedule, without setting money on fire

Once it works for one model, you want all five — and they don't behave the same. The big split:

Memory-only answers reflect training data: the slow, internet-wide accumulation of how often and how positively a brand appears. A new brand is invisible here for a long time.
Web-connected answers (Perplexity, ChatGPT search, Gemini, Grok-with-browsing) do live retrieval and will surface a brand that barely exists in training data — if its content ranks and reads well right now.

Tracking them as one blended number hides the most useful signal you have: the gap between your web-on and web-off visibility tells you whether your problem is content (fixable this month) or reputation (a longer game).

Two engineering notes that saved the budget:

Run engines concurrently, not sequentially. A scan is N questions × M engines × K samples; doing it serially is brutal latency. Fan out per engine and gather.
Web search is the cost bomb. Browsing-enabled calls cost multiples of a plain completion. Cap tool/search uses per call, cache aggressively on a freshness window (the web doesn't change in 10 minutes), and degrade to cheaper engines when you hit a budget ceiling. "Scan everything, every hour" will bankrupt you for zero added signal.

Bonus: citations are a free dataset, use them

When web-connected engines answer, they cite sources. Those citations are a literal list of the domains the model currently trusts for a topic. Extract and aggregate them across a category's scans and you get a ranked map of where to earn presence — the directories, comparisons, reviews, and threads the AI actually pulls from. I denormalize cited domains out of every result for this; it's one of the most-used outputs of the whole thing. Don't throw the citation array away — it's better targeting data than any keyword tool.

What this all rolls up to

If you take nothing else: AI search visibility is an observable, instrumentable system. Sample over time instead of once, control your inputs (brand-blind prompts), separate your signals (web-on vs web-off), resolve entities instead of grepping strings, and watch the trend rather than the draw. Do that and you get an honest number. Skip any one of them and you get a flattering lie.

I packaged all of this into a product called Zene — scheduled brand-blind scans across the five engines, entity-level mention resolution, per-engine trends, competitor comparison, and the cited-domain map — but the four problems above are the actual content. Whether you use the tool or roll your own, those are the traps.

One last, unrelated-but-related warning, since this is a builder crowd: the moment you have a domain, you'll get emails selling "1,000 forum backlinks" to boost this stuff. Don't. Those links land you next to replica-watch and pirated-movie spam, and no model — or search engine — trusts a brand keeping that company. The signal that moves AI visibility is the same boring, durable thing it always was: be genuinely worth quoting, and be corroborated by sources that aren't you.

Run a brand-blind scan on your own product. If the number's ugly, congrats — you found a real problem while it's still early enough to matter.