DEV Community: Lachezar Dimitrov

AI Visibility Tracking: What 23 AI Overviews Showed

Lachezar Dimitrov — Mon, 24 Aug 2026 11:58:00 +0000

AI visibility tracking means measuring whether AI answer engines mention and cite your brand when buyers ask questions in your category. Most tools report a mention count. That number alone is not actionable. In our own measurement of 23 buyer prompts, 91% triggered a Google AI Overview and we were cited in zero of the non-branded ones — and the reason turned out to be something a mention counter would never have shown us.

This is not a roundup of tools. We built a tracker, pointed it at our own category, and are publishing what came back, including the parts that make us look bad. If you are evaluating AI visibility tracking, the useful question is not which dashboard is prettiest. It is which number tells you what to do next.

What we measured, and how

On 26 July 2026 we ran 23 prompts through two surfaces, in the US market, in English. The prompt set was built the way a buyer actually searches: four branded prompts about our own company, and 19 non-branded prompts covering the questions our category gets asked — pricing, vendor selection, industry-specific automation, build-versus-buy.

Two signals, both machine-readable:

Google AI Overview. For each query: does an AI Overview trigger at all, is our domain among its cited references, and which domains are cited instead. We also captured our classic organic position for the same query, which turned out to matter more than we expected.
Embeddings retrieval. Whether our domain surfaces in the vector index layer that many AI applications query before generating an answer.

What we deliberately did not claim to measure: the answer text inside ChatGPT, Claude, or Perplexity. No API exposes what those systems said about your brand in a real user session. Any tool claiming to track this is running its own prompts through the API and inferring, which is a reasonable proxy but is not the same thing. We think vendors should say so plainly, so we are saying it about our own work first.

Finding one: AI Overviews are not an emerging surface, they are the surface

21 of 23 prompts (91%) returned a Google AI Overview. Not a handful. Not the informational ones only. Nearly every commercial question our buyers ask now gets answered above the organic results, by a system that cites a handful of sources and moves on.

That reframes what a ranking is worth. Position 4 on a query where an AI Overview occupies the first screen is not position 4 in any meaningful sense. If you are still reporting average position without segmenting by whether an AI Overview fires, your report is describing a page layout that most of your buyers are no longer seeing first.

Finding two: we scored zero, and the zero was the useful part

Here is the uncomfortable table. Of our four branded prompts, all four triggered an AI Overview and three cited us. Of our 19 non-branded prompts, 17 triggered an AI Overview and zero cited us.

A mention-counting tool would have reported "3 citations, 13% visibility" and drawn a line on a chart. That framing would have been actively misleading. The three citations were on queries containing our own company name — queries where a buyer already knows who we are. On every question where someone was actually shopping, we did not exist.

Branded visibility and non-branded visibility are different metrics measuring different business realities, and averaging them together produces a number that flatters you. If your AI visibility report shows a single blended percentage, split it. The branded number tells you whether your entity is understood. The non-branded number tells you whether you are in the consideration set. Only the second one is a growth metric.

Finding three: citation is mostly downstream of ranking

This is the finding that changed how we plan, and it comes from a second, smaller run rather than the study above. Two days later, on 28 July 2026, we re-tested six of the 19 non-branded queries. Four of them returned an AI Overview. For those four we captured every domain cited, then checked whether those same domains also ranked in the classic organic results for the identical query. So the overlap figures below describe four AI Overviews from 28 July, not the 21 from 26 July.

Across the 31 citations those four answers issued:

74% also ranked in the classic organic top 20
48% ranked in the top 10
Only 25% were cited without ranking anywhere in the first two pages

The sample is small and we would not present it as a universal law. But the direction is clear enough to act on: for most queries, Google is drawing its AI Overview citations from a pool it has already decided to rank. There is no separate side door. The bulk of "AI visibility optimization" is ordinary ranking work, and a tracker that reports citations without reporting your organic position for the same query has hidden the causal variable.

That 25% residual is real and worth chasing — it is where format, structure, and being the definitive source on a narrow question earn a citation without a ranking. But it is the exception, not the strategy. Anyone selling you an AEO methodology that bypasses ranking should be asked to show the overlap data.

Finding four: one query type ignored websites entirely

One prompt in our set asked for the top AI automation agencies in a specific city. The AI Overview cited exactly four domains, and every one of them was a directory: Clutch, DesignRush, GoodFirms, and The Manifest. No agency websites at all. Not the market leaders, not the best-optimized pages. Directories.

For location-shaped queries, Google appears to treat curated third-party lists as the authoritative source and does not read vendor sites for the answer. If your visibility strategy for local intent is on-page content, you are optimizing an asset the system is not consulting. The lever is a complete, claimed directory profile.

We found this because we measured. We would not have found it by reading a best-practices article, and a mention-count dashboard would have shown a zero with no explanation attached.

Finding five: what actually gets cited

Back to the 26 July study. Across its 21 AI Overviews, 149 citations — counting each domain once per answer, out of 174 raw references — went to 116 unique domains. The concentration is lower than we expected — this is not a cartel of five big publishers.

User-generated and video content accounted for 27 of those 149 citations, about 18%. YouTube appeared in 11 of the 21 answers, Reddit in 8, LinkedIn in 4, Medium in 3, Quora in 1. That is a meaningful minority but not the majority, which surprised us: the common claim that AI Overviews are dominated by Reddit did not hold in our category.

The remaining 82% was ordinary vendor and agency content. Which is the encouraging half of the result. The citation pool is not closed. It is populated by companies that rank, and ranking is a solvable problem.

What we would actually track

Based on running this, here is what belongs on an AI visibility report and what does not.

Track these:

AI Overview trigger rate across your prompt set. This tells you how much of your category has moved above the fold. Ours is 91%; if yours is 20%, your priorities are completely different from ours.
Branded and non-branded citation rates, reported separately. Never blended.
Citation-to-ranking overlap. For every query where you are not cited, your own organic position for that query. This single column converts "we are invisible" into either "we need to rank" or "we rank and still are not cited," which are entirely different problems with entirely different fixes.
The competitor citation map. Which domains are being cited instead of you, and how often. This is your real competitive set in AI answers, which frequently differs from your assumed competitive set.

Be sceptical of these:

A single blended "AI visibility score." It compresses away every decision-relevant distinction.
Claims to read actual ChatGPT or Claude conversation answers. No public API exposes them.
Sentiment analysis on AI mentions, before you have any mentions to analyse. Sequence matters.

What this cost

The full 23-prompt run costs roughly ten cents in SERP API credit, plus the retrieval queries. It is a Python script that writes dated JSON, so month-over-month comparison is a diff rather than a screenshot.

We are not arguing nobody should buy a tool. Commercial trackers offer prompt libraries, scheduling, and reporting that a script does not. But you should know that the underlying measurement is cheap, and price the subscription against convenience rather than against access to otherwise-unavailable data.

How this connects to the work

We publish this partly because it is the same discipline we apply to client systems. In our AI consulting engagements the first deliverable is usually a measurement baseline, because a workflow nobody measured cannot be improved and an AI opportunity nobody sized cannot be prioritized. The audit that tells you where AI should not be applied is as valuable as the one that tells you where it should.

The same logic applies here. We ran this study on ourselves before recommending the approach to anyone, found that we score zero on every commercial query in our own category, and published it. If you are weighing whether you need outside help with AI strategy at all, our guide to what an AI strategy consultant does and when you need one covers the decision honestly, including when the answer is that you do not.

Frequently asked questions

What is AI visibility tracking?

Measuring whether AI answer engines — Google AI Overviews, ChatGPT, Perplexity, Claude — mention and cite your brand when people ask questions in your category. Useful tracking captures three things: whether an AI answer appears at all, whether you are cited, and which domains are cited instead of you. Tracking that reports only a mention count leaves out the diagnostic information.

Can any tool actually see what ChatGPT says about my brand?

Not in real user sessions. No public API exposes what a given assistant told a given user. Tools that report this are running their own prompts through the API and treating the results as a sample. That is a legitimate proxy and often a useful one, but it is a sample of what the model tends to say, not a record of what users were told. Be wary of any vendor that blurs the distinction.

Is AI visibility a separate discipline from SEO?

Mostly not, on our data. In a follow-up run on 28 July 2026 — separate from the 23-prompt study of 26 July — we re-tested six of our non-branded queries; four returned an AI Overview, and across the 31 citations those four issued, 74% went to domains already ranking in the classic organic top 20 for the same query, and 48% to the top 10. On a sample that small we would not call it a law, but it points one way: the majority of AI visibility work is ranking work. A real but minority share of citations goes to pages that do not rank, which is where answer-shaped formatting and topical depth earn their keep.

How often should we re-measure?

Monthly is enough for most businesses, run on the same day each month with an unchanged prompt set so the comparison is meaningful. Changing prompts between runs is the most common way these reports become uninterpretable. Re-measure sooner only after a significant content or technical change you want to attribute.

We are cited on branded queries but not commercial ones. What does that mean?

That your entity is understood but you are not in the consideration set — exactly the pattern we found in our own data. It means the fix is not entity or schema work, which is already succeeding. It is competing for the commercial queries themselves: ranking for them, earning references from sources the answer engines trust, and being present wherever those queries are answered by third-party lists rather than vendor sites.

What is the cheapest way to start?

Write 20 prompts your buyers would actually type, split branded and non-branded, and check them manually once. It takes an afternoon and will tell you your trigger rate and roughly where you stand. Automate it only once you know the measurement is telling you something you will act on.

Measure before you optimize

We went into this expecting to learn which AI platforms mattered most. We came out with something more useful and less comfortable: a zero on every commercial query, a clear causal explanation for the zero, and one query type where the entire answer came from directories we had not claimed.

None of that came from a dashboard. It came from running the measurement, keeping the raw output, and being willing to publish the bad number. If you want to talk through what this would look like for your category, book a strategy call and bring the questions your buyers actually ask.

Originally published at atilab.io. ATI is an AI automation agency; we publish what our own implementation work measures, including the results that make us look bad.

Build Your Own AI Visibility Tracker (Without a Subscription)

Lachezar Dimitrov — Fri, 14 Aug 2026 07:05:01 +0000

Originally published on the ATI Lab blog. Full context, including our 23-prompt AI Overview study, is there.

You can build a working AI visibility tracker in about 200 lines of Python. It polls Google AI Overviews for a fixed prompt set, records which domains get cited, captures your own organic position on the same query, and writes dated JSON so months compare cleanly. Our 23-prompt run cost $0.092 in API credit. The commercial tools that rank for this term start at $99–$165 a month. Here is the build, and an honest account of what it cannot do.

This guide is written from our own tracker, which we run against atilab.io. The code below is the code we actually execute — not pseudocode. Research and drafting were AI-assisted; every number is from our own run files or a source linked in-line, and a human checked each one.

What does an AI visibility tracker actually have to measure?

Three things, and most dashboards report only the first:

Does an AI answer appear at all for the query? Trigger rate is the denominator for everything else. If AI Overviews fire on 20% of your category's queries, this whole exercise is a side project. In ours they fired on 21 of 23.
Are you cited? Split branded from non-branded and never blend them. Branded citations tell you your entity is understood. Non-branded citations tell you whether you are in the consideration set. Only the second is a growth metric.
Where do you rank in classic organic on that same query? This is the column almost nobody captures, and it is the one that converts "we are invisible" into a specific instruction. More on why below.

Everything else — sentiment, share-of-voice indices, a single blended visibility score — is a presentation layer over those three fields. You can add it later. You cannot add the third field later, because it has to be captured in the same request, on the same day, against the same SERP.

What does it cost to run yourself versus buy?

Our numbers, from the run files. The Google signal uses DataForSEO's live advanced SERP endpoint, which returned an API-reported cost of $0.0040 per query on every call in our run. Twenty-three prompts is $0.092. Run it monthly and the year costs about $1.10.

For comparison, published pricing on the tools currently ranking for "ai visibility tracker" (checked 12 August 2026): Rankscale lists Pro at $99/month for 1,200 credits, Growth at $385/month, Enterprise at $780/month. Semrush bundles AI visibility into its main plans — Starter at $165.17/month billed annually, with 50 prompts tracked daily, rising to 200 daily prompts on Advanced at $455.67/month.

Those are not equivalent products and we are not going to pretend they are. The subscriptions sample answers from ChatGPT, Gemini, Perplexity, Claude and others, schedule the runs, store history, and produce reports somebody else maintains. The DIY build covers two surfaces and produces a JSON file. What the cost comparison establishes is narrower and still useful: the underlying measurement is cheap. Price the subscription against convenience and coverage, not against access to data you could not otherwise get.

Step 1: build the prompt set — this decides everything downstream

The single biggest determinant of whether your tracker is useful is the prompt set, and it is the part no tool can do for you. Ours is 23 prompts: 4 branded, 19 non-branded, tagged by funnel stage and by the page each one should ideally send traffic to. Each record carries two forms of the same intent:

{"id": 6, "type": "non-branded", "funnel": "MOFU", "pri": "P1",
 "target": "/ai-agent-cost",
 "prompt": "What does it cost to build and run AI agents for a business?",
 "query": "how much does it cost to build ai agents for business"}

The prompt is the conversational form a person types into an assistant. The query is the search-shaped form you send to Google. They are different strings on purpose, and collapsing them into one field is the most common way these projects produce uninterpretable data.

Two rules that matter more than the prompt wording:

Keep the ids stable forever. The value of this tracker is the diff between months. Change a prompt and its history is worthless; add prompts at the end with new ids instead.
Tag each prompt with the page it should serve. When a prompt shows an AI Overview and you are absent, the target field tells you instantly which page has failed, rather than starting a research project.

Step 2: query the AI Overview and capture the references

One request per query. The parameter that matters is load_async_ai_overview — without it the AI Overview block is frequently missing from the response even when it fires on the live SERP, and you will conclude your trigger rate is low when it is not.

res = dfs.call("serp/google/organic/live/advanced", [{
    "keyword": p["query"],
    "location_code": 2840,          # US
    "language_code": "en",
    "depth": 20,
    "load_async_ai_overview": True,
}])

Then walk the items once, pulling both signals out of the same response:

for item in res[0].get("items") or []:
    itype = item.get("type") or ""
    if itype == "ai_overview" and not out["aio_present"]:
        out["aio_present"] = True
        refs = item.get("references") or []
        out["aio_refs"] = [host_of(r.get("url") or "") or (r.get("domain") or "")
                           for r in refs]
        out["aio_cites_brand"] = any(BRAND in d for d in out["aio_refs"])
    elif itype == "organic" and out["organic_rank"] is None:
        if BRAND in (item.get("domain") or ""):
            out["organic_rank"] = item.get("rank_absolute")

Note the fallback on the reference: some entries carry a url, some carry only a domain. Read one field and you will silently drop citations.

Why capturing your organic rank in the same request changes the report

Because citation is mostly downstream of ranking. In our study we took six non-branded queries where an AI Overview fired, captured all 31 citations, and checked each cited domain against the classic results for the identical query: 74% also ranked in the organic top 20, 48% in the top 10, and only 25% were cited without ranking on the first two pages. Small sample, and we would not present it as a universal law — but the direction is clear enough to plan against. Google is largely drawing its citations from a pool it has already decided to rank. A tracker that reports citations without reporting your position on the same query has hidden the causal variable, and its output is a number you cannot act on.

With both fields, every row falls into one of four cells, and each cell has a different instruction attached:

That distribution is the reason we are unsentimental about AEO tactics. Nineteen of twenty-three prompts told us the same thing: rank first. We published the full findings in our measurement of 23 AI Overviews in our category, including the queries where Google ignored vendor sites entirely.

Step 3: add a retrieval signal

The second surface worth polling is the embeddings layer that many AI applications query before generating an answer. We use Exa: send the conversational prompt form, and record whether your domain comes back and at what rank.

def check_exa(p, api_key):
    out = {"exa_rank": None, "exa_url": None, "exa_top": []}
    res = exa_search(p["prompt"], api_key)
    for i, r in enumerate(res.get("results") or [], 1):
        if BRAND in host_of(r.get("url")):
            out["exa_rank"] = i
            break
    return out

This signal behaves very differently from the Google one and that is the point of having it. We surfaced in retrieval on 5 of 23 prompts, at ranks 1, 1, 1, 1 and 6 — four first places on a set where Google cited us three times, all branded. Retrieval visibility and citation visibility are not the same thing, and a tracker with one surface will tell you a confident, incomplete story.

Step 4: count domains per prompt, not per reference

A small decision with a large effect on the competitor map. One AI Overview citing five pages of the same domain is still one prompt's worth of visibility. Count it once:

freq = {}
for r in rows:
    for d in set(r.get("aio_refs") or []):   # set() — one prompt, one vote
        freq[d] = freq.get(d, 0) + 1

Skip the set() and any domain with a habit of multi-page citation looks like it owns your category. With it, our map reads cleanly: YouTube appeared in 11 of the 21 triggered AI Overviews, Reddit in 8, LinkedIn in 4. Counted as raw references those three domains are 37 of 174 citations, 21%. That is a meaningful minority, not the domination the common advice implies — the remaining 79% went to ordinary vendor and agency content, spread across 116 distinct domains. The citation pool is not closed, which is the encouraging half of the result.

Step 5: write dated output, never overwrite

The whole value of this thing is the month-over-month diff, so the output directory is the date and nothing overwrites anything:

outdir = os.path.join(OUTDIR, datetime.date.today().isoformat())
os.makedirs(outdir, exist_ok=True)
json.dump({"date": date, "location_code": loc, "language_code": lang,
           "brand": BRAND, "results": rows},
          open(os.path.join(outdir, "results.json"), "w"), indent=1)

Write a flat summary.csv alongside it. JSON for diffing, CSV for the person who wants to sort it in a spreadsheet.

What broke when we ran it

Four things, all worth knowing before you spend an afternoon debugging them:

Transient SERP API errors. The provider intermittently returns an internal server error on an otherwise valid task. Wrap every call in a retry with backoff, and return None rather than crashing the run — losing one query is fine, losing the other 22 is not.
Missing AI Overview blocks. Covered above: without the async-load parameter your trigger rate will be understated.
Reference objects with no URL. Read url first and fall back to domain.
Location and language drift. Pin them explicitly in the code, not in a config you might edit. A run at a different location code is a different study, and if it lands in the same series you will spend a month explaining a trend that is an artefact.

def call_dfs(path, payload, attempts=3):
    for n in range(attempts):
        try:
            return dfs.call(path, payload)
        except Exception as exc:
            if n == attempts - 1:
                return None
            time.sleep(2 * (n + 1))

What this build cannot measure

Be straight about this, because the vendors ranking above this page are not always. No public API exposes what ChatGPT, Claude, Gemini or Perplexity actually said to a real user in a real session. Every tool reporting "your ChatGPT visibility" is running its own prompts through an API and treating the output as a sample. That is a legitimate proxy and often a useful one. It is a sample of what a model tends to say, not a record of what your buyers were told.

You can build that proxy yourself too — send your prompt list to an assistant API, check whether your domain appears in the answer, repeat n times per prompt because the outputs vary. Just cost it honestly: a meaningful sample means several runs per prompt per platform, and that is where the DIY approach stops being nine cents and starts approaching a subscription. This is the point at which buying is a reasonable decision.

How often should you run it, and what do you do with the output?

Monthly, on the same day, with an unchanged prompt set. More frequent runs mostly measure SERP volatility. Re-run early only after a specific content or technical change you want to attribute.

Then read the report as a work queue, not a scoreboard. Rows in the bottom-right go to ordinary ranking work. Rows in the top-right — you rank, you are not cited — go to a formatting pass on an existing page, which is the cheapest work on the list. Rows in the top-left get a diary note and a re-check. That is the whole method, and it is the same discipline we apply in AI consulting engagements, where the readiness audit exists to say where AI should not be applied as much as where it should. A measurement baseline first; a build only where the baseline says one is justified. We wrote about the same trap on the delivery side in measuring AI agents on engineering teams — the metric that is easy to collect is rarely the one that changes a decision.

Frequently asked questions

What is an AI visibility tracker?

A tool that measures whether AI answer engines mention and cite your brand when people ask questions in your category. A useful one captures three fields per query: whether an AI answer triggered at all, whether you were cited, and which domains were cited instead. Adding your classic organic position for the same query turns those observations into an instruction.

Do I need to code to track AI visibility?

No — commercial tools start around $99–$165 a month and cover more surfaces than a script does. Building it yourself is worth it when you want the raw data, control of the prompt set, and month-over-month files you own. The build described here is a single stdlib Python file plus one SERP API account.

How much does a DIY AI visibility tracker cost to run?

Our 23-prompt Google run cost $0.092 in API credit, at an API-reported $0.0040 per query, plus retrieval-API usage on its own plan. Run monthly, that is a little over a dollar a year for the Google signal. The real cost is the hour you spend building the prompt set properly.

Can a tracker see what ChatGPT says about my brand?

Not in real user sessions. No public API exposes what an assistant told a specific user. Tools reporting this are sampling their own prompts through an API, which is a reasonable proxy that should be labelled as one.

Is AI visibility a separate discipline from SEO?

Mostly not, on our data: 74% of the AI Overview citations we traced also ranked in the classic organic top 20 for the same query. A real but minority share of citations goes to pages that do not rank — that is where answer-shaped formatting earns its keep. Treat AEO as a formatting and structure layer on top of ranking work, not as a replacement for it.

How many prompts should I track?

Twenty to thirty is enough to be diagnostic and small enough that you will actually keep it current. Weight it towards non-branded commercial questions, tag each prompt with the page that should serve it, and never change a prompt once it has history.

Start with the measurement

The reason we built this rather than bought it was not the $99. It was that we wanted the raw rows, including the ones that made us look bad — and on our first run, nineteen of them did. If you would like a read on where AI visibility sits against the rest of your growth priorities, book a strategy call and we will go through your own numbers rather than ours.