Why your AI-visibility report should tell you when the fix didn't work

CitePack — Fri, 24 Jul 2026 11:30:25 +0000

To know whether a change actually improved your AI citations, measure the same query on the same engine before and after — and do it more than once, because AI engines are non-deterministic. Ask ChatGPT or Perplexity the identical question twice and you can get different cited sources each time. A single "after" that looks better than your "before" isn't proof. It might just be the engine rolling different dice.

This is the part almost every AI-visibility tool gets wrong. Their dashboards only go up. Here's why that should worry you, and what measuring it honestly actually takes.

Why one probe is a coin flip

AI answer engines don't return the same sources every time, even for the identical prompt. Same-day, back-to-back runs routinely cite different pages. So if you check your citation once before a fix and once after, you haven't measured your fix — you've measured your fix plus a large, invisible pile of randomness, and you can't tell which one moved the number.

The fix: probe several times, build an interval

The honest version is boring and it works: probe each engine several times per query (we use seven or more), then express the result as a rate with a confidence interval instead of a single yes/no. A citation that shows up on 6 of 7 runs is a real signal. One that shows up on 1 of 7 is noise wearing a costume. The interval is what lets you tell them apart.

The step nobody shows you: the noise floor

Here's the move that separates measurement from theater. Before you trust that a change caused an improvement, run the exact same test on a site where nothing changed — a control. Measure how often the "result" moves anyway. That's your noise floor: the rate at which the engines flip their answer for no reason at all.

If your tool can't tell you its own noise floor, it cannot tell a real win from a random one. It just reports every upward wiggle as a victory and hopes you don't re-check. We run this control test on our own site continuously, precisely so we can say — with a straight face — when a customer's change beat the noise and when it didn't.

"Still not cited" is a real result

Most tools have no state for "the fix didn't work." Ours does, and it's first-class. If your pages went live, the engines re-indexed, and the citation still didn't move, the report says exactly that: no change yet. That's not a bug or a hedge. A tool that can only ever tell you good news isn't measuring anything.

How to do this yourself, no tool required

Pick one engine and one question a buyer would actually ask.
Ask it five to seven times. Note whether your domain appears in the cited sources each time.
Write down the rate (e.g. "cited 4/7").
Ship your change. Wait for re-indexing — usually a couple of weeks.
Repeat the exact same probe. Compare the rates, not single runs.

Do that and you're already ahead of most paid dashboards.

Where a tool earns its keep

Everything above is doable by hand; it's just tedious and easy to fool yourself with. A tool earns its price by automating the repeated probes, computing the interval, running the control, and — the part we care most about — wiring every recommended fix to the specific number that will prove or disprove it on re-check. That's what turns a checklist into a diagnosis.

If you want to see your own four-engine board — where you're cited, where you're only mentioned, and where you're invisible — the scan is free at citepack.com.

I built an AI citation checker. Two of four engines think it's a Python module.

CitePack — Sat, 18 Jul 2026 20:11:43 +0000

To check whether an AI engine cites your brand, ask each engine (ChatGPT, Perplexity, Claude, and Google's Gemini) your top buyer question, then look at whether your domain appears in the sources it used to answer. They disagree constantly. I run this probe on my own company every week, and this week two of the four didn't even know my company is a company.

Here's the unedited board for "What is CitePack?" on 16 July 2026:

Perplexity: cites us. Correct.
Google (Gemini): cites us. Correct.
ChatGPT: describes a Python module that doesn't exist.
Claude: describes the wrong entity entirely.

Two out of four. And on the query that actually converts, "best tool to check if AI cites my site," I'm cited by none of them. The engines name Frase, Semrush, and a Reddit thread. Not me.

Why two engines cite us and two don't

The split isn't random. Perplexity and Gemini retrieve live from the web at answer time, so they picked up the entity signals we've published — the site, the LinkedIn company page, the structured data. ChatGPT and Claude lean more heavily on training data and weaker retrieval, so when they hit an unfamiliar name like "CitePack," they pattern-match: "Cite" + "Pack" sounds like a developer tool, so they invent one.

The core lesson: getting cited by AI is mostly an entity-recognition problem, not a content problem. If the engine doesn't know who you are, no amount of great content earns the citation, because it can't attribute it to you.

The fix (the same one I sell)

There's no trick here, and nothing that coerces a model. You make the real entity unmissable:

A single, factual knowledge-base anchor (a Wikidata item) that feeds Google's Knowledge Graph and the retrieval layer the other engines lean on.
Consistent, claimed profiles that corroborate each other (Crunchbase, Product Hunt, G2, LinkedIn), all describing the same entity the same way.
Answer-first pages on the exact questions buyers ask, structured so the engine can lift a clean, attributable answer.

Then you re-probe, and you watch ChatGPT flip from "Python module" to "the company." That flip is the receipt.

Why I'm showing you the misses

Because a citation you can't prove is just a claim. The product I'm building, CitePack, exists to end that: it measures whether the engines cite you, ships the deploy-ready fix, then re-tests the real engines and hands you the dated before → after. If I won't run that loop on myself in public, including the 0/4, I have no business selling it.

So consider this the "before." I re-probe on 17 July and publish the "after," whichever way it goes.

If you want to see your own four-engine board, the scan is free at citepack.com. It'll probably surprise you which engines already cite you, and which think you're something you're not.

DEV Community: CitePack