To know whether a change actually improved your AI citations, measure the same query on the same engine before and after — and do it more than once, because AI engines are non-deterministic. Ask ChatGPT or Perplexity the identical question twice and you can get different cited sources each time. A single "after" that looks better than your "before" isn't proof. It might just be the engine rolling different dice.
This is the part almost every AI-visibility tool gets wrong. Their dashboards only go up. Here's why that should worry you, and what measuring it honestly actually takes.
Why one probe is a coin flip
AI answer engines don't return the same sources every time, even for the identical prompt. Same-day, back-to-back runs routinely cite different pages. So if you check your citation once before a fix and once after, you haven't measured your fix — you've measured your fix plus a large, invisible pile of randomness, and you can't tell which one moved the number.
The fix: probe several times, build an interval
The honest version is boring and it works: probe each engine several times per query (we use seven or more), then express the result as a rate with a confidence interval instead of a single yes/no. A citation that shows up on 6 of 7 runs is a real signal. One that shows up on 1 of 7 is noise wearing a costume. The interval is what lets you tell them apart.
The step nobody shows you: the noise floor
Here's the move that separates measurement from theater. Before you trust that a change caused an improvement, run the exact same test on a site where nothing changed — a control. Measure how often the "result" moves anyway. That's your noise floor: the rate at which the engines flip their answer for no reason at all.
If your tool can't tell you its own noise floor, it cannot tell a real win from a random one. It just reports every upward wiggle as a victory and hopes you don't re-check. We run this control test on our own site continuously, precisely so we can say — with a straight face — when a customer's change beat the noise and when it didn't.
"Still not cited" is a real result
Most tools have no state for "the fix didn't work." Ours does, and it's first-class. If your pages went live, the engines re-indexed, and the citation still didn't move, the report says exactly that: no change yet. That's not a bug or a hedge. A tool that can only ever tell you good news isn't measuring anything.
How to do this yourself, no tool required
- Pick one engine and one question a buyer would actually ask.
- Ask it five to seven times. Note whether your domain appears in the cited sources each time.
- Write down the rate (e.g. "cited 4/7").
- Ship your change. Wait for re-indexing — usually a couple of weeks.
- Repeat the exact same probe. Compare the rates, not single runs.
Do that and you're already ahead of most paid dashboards.
Where a tool earns its keep
Everything above is doable by hand; it's just tedious and easy to fool yourself with. A tool earns its price by automating the repeated probes, computing the interval, running the control, and — the part we care most about — wiring every recommended fix to the specific number that will prove or disprove it on re-check. That's what turns a checklist into a diagnosis.
If you want to see your own four-engine board — where you're cited, where you're only mentioned, and where you're invisible — the scan is free at citepack.com.
Top comments (0)