DEV Community

Cover image for AI Brand Tracking: A Practical Check You Can Run Today
Sushi
Sushi Subscriber

Posted on AI-assisted

AI Brand Tracking: A Practical Check You Can Run Today

Someone asks ChatGPT for "the best fragrance-free sunscreen for oily skin". The answer names three brands, links two review sites and gets one price wrong. If one of those brands is yours, or should have been, you'll never hear about it. Nothing shows up in your analytics.

AI brand tracking is how you find out. You ask the assistants the questions your buyers ask, you record what they say, and you check it on a schedule. It sounds like a marketing job, but it's mostly a data-collection problem, so it's a good one for a developer to own.

Here's a version you can set up in an afternoon and keep up in about an hour a month.

Why AI brand tracking matters for developers

Three reasons it ends up on your desk:

  • The fix is usually on your site. When an assistant gets a fact wrong, the cause is often an outdated page, a missing spec, or structured data that says something different from the visible text. Those are code changes.
  • There's no dashboard for it. Assistants don't publish what people ask them or how often you appear. You have to generate the data yourself.
  • Answers vary from run to run. Ask the same question twice and you can get a different list. One screenshot proves nothing; you need repeated runs logged the same way each time.

Step 1: Pick the brand questions to test

Write 15 to 20 questions in your buyers' words, not your marketing copy. Split them into four groups, because each one tests something different:

Group What it tests Example (skincare)
Category Are you recommended at all? "Best fragrance-free sunscreen for oily skin in India"
Comparison How are you described next to a rival? "[Your brand] vs [competitor] sunscreen"
Brand Are the facts about you right? "Is [your brand] sunscreen safe for sensitive skin?"
Problem Do you appear before they know the product? "Why does my sunscreen leave a white cast?"

Good sources for the wording: your support inbox, product reviews, Reddit threads in your category, and Google's "People also ask" boxes.

Keep the list fixed. If you change the questions every month, you can't compare the results.

Step 2: Run the same prompts across LLMs and record the answers

You have two ways to collect answers, and you want both.

In the apps (what shoppers actually see). Open ChatGPT, Gemini, Perplexity and Google, use a fresh chat each time, and sign out or use a private window where you can, so your history doesn't shape the answer. Paste each question and log the result. This is the version shoppers see, with web search switched on.

Through the APIs (repeatable, scriptable). API answers often differ from the apps, because the app may search the web and the API call usually doesn't. Even so, they're useful for spotting what a model "knows" about you without a search step, and you can run them as often as you like.

Here's a small script for the API side. It uses OpenRouter so one key covers several model providers, and it only needs the Python standard library. Copy the exact model IDs you want from openrouter.ai/models.

import csv, json, os, re, time, urllib.request
from datetime import date

BRAND = "YourBrand"
COMPETITORS = ["Competitor A", "Competitor B"]
MODELS = ["provider/model-one", "provider/model-two"]  # exact IDs from openrouter.ai/models
QUESTIONS = [
    "Best fragrance-free sunscreen for oily skin in India",
    "Is YourBrand sunscreen safe for sensitive skin?",
]
RUNS = 2  # answers vary, so ask each question more than once

def ask(model, question):
    request = urllib.request.Request(
        "https://openrouter.ai/api/v1/chat/completions",
        data=json.dumps({"model": model, "messages": [{"role": "user", "content": question}]}).encode(),
        headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}",
                 "Content-Type": "application/json"},
    )
    with urllib.request.urlopen(request, timeout=120) as response:
        return json.load(response)["choices"][0]["message"]["content"]

def named(text, name):
    return re.search(rf"\b{re.escape(name)}\b", text, re.IGNORECASE) is not None

rows = []
for question in QUESTIONS:
    for model in MODELS:
        for run in range(1, RUNS + 1):
            answer = ask(model, question)
            rows.append({
                "date": date.today().isoformat(),
                "question": question,
                "model": model,
                "run": run,
                "brand_named": named(answer, BRAND),
                "competitors_named": "; ".join(c for c in COMPETITORS if named(answer, c)),
                "urls_cited": "; ".join(sorted(set(re.findall(r"https?://[^\s)\]>\"']+", answer)))),
                "accurate": "",  # fill in by hand: yes / no / partly
                "notes": "",     # what was wrong, and where it probably came from
                "answer": answer,
            })
            time.sleep(1)

with open(f"ai-brand-check-{date.today()}.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=rows[0].keys())
    writer.writeheader()
    writer.writerows(rows)

for model in MODELS:
    answers = [r for r in rows if r["model"] == model]
    print(f"{model}: {BRAND} named in {sum(r['brand_named'] for r in answers)} of {len(answers)} answers")
Enter fullscreen mode Exit fullscreen mode

For the app checks, log into the same columns by hand so both sources land in one sheet:

date | question | assistant | run | brand_named | competitors_named | urls_cited | accurate | notes
Enter fullscreen mode Exit fullscreen mode

A plain-text template helps keep the manual runs consistent. Paste the question exactly as written, then fill this in straight after:

Question:          [paste exactly]
Assistant:         ChatGPT / Gemini / Perplexity / Google AI
Signed out:        yes / no
Brand named:       yes / no   Position in answer: 1st / 2nd / later / not named
What it said:      [one line]
Correct:           yes / no / partly   What was wrong: [ ]
Sources cited:     [domains]
Enter fullscreen mode Exit fullscreen mode

Step 3: Score accuracy, mentions, and sources cited

Three numbers per assistant are enough to start:

  • Mention rate: the share of answers that name you. Report it as "named in 5 of 8 runs", not a ranking. There's no position 1 in a generated answer.
  • Accuracy: of the answers that name you, how many get the price, ingredients, sizes, shipping and returns right.
  • Sources: which domains get cited, and how often. This is the most useful column. It tells you which review sites, forums and buying guides the assistant trusts in your category.

Sort the sources column by count. If one buying guide shows up in half the answers and you're not in it, you've found your next outreach target.

Step 4: Fix the gaps on your site and re-check

You can't edit an AI answer. You change the pages it reads, and the answer follows: sometimes within days, sometimes months later.

What the sheet shows Likely cause Fix
A wrong fact about your product An old page, a stale marketplace listing, or a detail missing from your site Correct it on your product page and every listing, in the same words
Your structured data disagrees with the page Product schema generated from an old field Make the JSON-LD and the visible text come from one source
You're named but never linked The model knows the brand but didn't read a page Make sure the matching page states the answer in plain text, high up
A competitor wins with one cited guide That guide is the assistant's main source Pitch the guide's author with real product details
A discontinued product shows up The old URL still resolves or is still listed elsewhere Redirect it to the replacement and update your feeds

Then run the same questions next month and compare the CSVs. Same questions, same columns, same number of runs. That's the whole trick.

A few honest limits: twenty questions are a sample, not the thousands of ways people ask. API results aren't what shoppers see in the apps. And a mention isn't a sale, so pair this with referral traffic from assistants and searches for your brand name.

Read the full walkthrough here: https://wolfvisibility.com/blog/ai-brand-monitoring-ecommerce

Top comments (0)