DEV Community: Robin

One AI citation check is a sample: how much answers change between identical runs, and how to log so you can tell a real move

Robin — Tue, 13 Oct 2026 08:58:35 +0000

You ask ChatGPT a buyer question on Monday and your site is cited. On Thursday you ask the exact same question and it is gone. Did you lose something, or did you just roll the dice twice?

Three studies published in the past week let us put rough numbers on that, and the short version: one check tells you very little.

What the new data says about repeat runs

Kingy AI Citation Index, Wave One (Curtis Pyke, published 5 October 2026). The study asked OpenAI, Claude and Gemini the same 1,200 designed English questions twice through their official APIs, 7,200 completed answers in total. It compared cited domains between runs with Jaccard overlap (shared domains divided by all domains in either run).

Same provider, two runs: 0.664 for OpenAI, 0.515 for Claude, 0.319 for Gemini.
Different providers, same question: about 0.06 for every pair.
Answers that had any native citation at all: OpenAI 71.6%, Claude 53.7%, Gemini 25.2%.

So even asking the same API the same question in the same collection window, a third or more of the cited domain set typically changes, and on Gemini roughly two thirds. Caveats the author states: the panel was designed for the study, not sampled from real user demand; API results can differ from the consumer apps; two observations in one window cannot describe future behavior. Kingy.ai commissioned and funded it.

Is My Brand in AI, AI Answer Evidence Index round two (Minel Gunesoglu, published 8 October 2026). Fifty price and cost questions went to five engines. The first ten were asked a second time on 7 October to Google AI Overviews, ChatGPT and Gemini. On questions where both answers cited a page, Google AI Overviews cited again 32 of the 51 pages from its first answers, ChatGPT 5 of 17, and Gemini 17 of 29. Single questions swung hard: one ChatGPT answer about local moving prices went from 5 of 6 figures found on its cited pages to 0 of 6. The study's own reading: "One answer is one draw." Caveats: ten repeated questions, English, New York location, and the author sells an AI visibility product.

cloro AI Search Index (freshness check dated 7 October 2026). This one is about a step before "am I cited": does the engine cite anything at all? On cloro's production monitoring corpus, Gemini attached a source to 68.2% of answers in the 30 days to 7 October. ChatGPT was 99.6% in July, 79.6% in the 30 days to 19 August and 97.6% in the 30 days to 7 October. Google AI Mode slid from 99.4% to 90.7% to 84.5%. cloro's own line: no single figure for those engines "survives long enough to plan around". cloro sells a search data API, and notes its own two methods disagree (Gemini 32.9% against 49.5% in the same July fortnight).

So a missing citation can mean the engine gave no citations that time, or picked a different handful of sources from a pool that changes on every run, or genuinely stopped using you. One check cannot tell those apart.

A logging routine that can tell a real move

1. Freeze your prompts. Pick five buyer questions written the way a customer would ask, without your brand name. Never edit the wording.

2. Run each prompt at least twice per engine per session, in fresh chats. Two is the minimum; three is better.

3. Log more than "cited yes or no". One row per run, with these cells:

date, engine, prompt id, run number
answer had any citation (yes/no)
named (your brand appears in the text)
cited (your URL is in the sources)
figures correct (if the answer states your price or a limit, is it right?)

The "any citation" cell is the one most people skip. If Gemini gave no sources at all, you did not lose a citation. There was none to win that time.

4. Report rates, not single results. "Cited in 4 of 10 ChatGPT runs this month" is a measurement. "Cited on Tuesday" is an anecdote.

5. Only call it a move when it holds. My rule: a change counts when it shows up in at least two fresh runs in two separate sessions, and in the same direction.

6. Keep engines in separate columns. Kingy found about 6% domain overlap between providers. A blended "AI score" averages things that barely share sources.

The part citation logs cannot answer

Even a perfect citation log only measures visibility, not whether anyone clicked.

Bluehost's parent company Newfold Digital put a number on that gap. In a Search Engine Journal podcast recap (Loren Baker with CEO Sachin Puri, 7 October 2026), a 90 day Bluehost analysis counted 83.4 million verified AI crawler requests per day across its hosted sites: roughly 90% for model training, about 9% for search indexing, about 1% for live assistant fetching. For OpenAI, Bluehost estimated one observable referral visit per 5,350 verified HTML crawl requests, and Puri says that is not a conversion rate. His proposed framework is three separate measurements: visibility, accuracy, action.

Attribution is getting slightly better. Search Engine Journal (Roger Montti, 1 October 2026) reported that Gemini has started adding UTM parameters to outgoing links; Google's John Mueller replied "Nice. I see it too." Google has not documented when the tags are added. A Reddit commenter quoted in the piece reported that the Gemini mobile app passes a referrer only around 9 percent of the time, which is a community report, not a measured figure.

So the practical next step after a citation log:

In your analytics, filter landings by referrer and by utm_source for the AI engines, and keep both views, because neither catches everything.
Log AI visits, sign ups and sales as their own rows, separate from citations.
Add a "How did you first hear about us?" field to your sign up or checkout, because some AI influenced visitors arrive as Direct and only their own answer tells you.
Do not read crawler hits in your logs as discovery.

What this can't tell you

None of these studies measured your niche. Kingy and Is My Brand in AI used APIs or scripted sessions, not a logged in person in the app. Every source above sells something related to AI visibility or hosting. Your own five prompts are a small sample too: 2 of 10 to 4 of 10 can be noise. The log does not make you cited. It makes you less likely to rewrite a good page because of one unlucky run.

A question for you

If you already log AI citations: how many repeat runs do you do before you trust a change, and have you ever seen a citation shift line up with an actual change in referral visits? I have not found a clean match yet on small sites and would like to compare notes.

Two things I made for this (disclosure: these are my own products).

The 5-Prompt Citation Snapshot (free): a starter log for five frozen prompts across engines, with separate cells for named and cited. https://normbrytande.gumroad.com/l/quwzbl
The AI Referral Ledger ($19): a 30 day, files only kit (11 page PDF, a ledger CSV, copy paste "How did you first hear about us?" field text and a weekly checklist) for checking whether AI engines actually send visits, sign ups or sales. Every AI attributed row gets one label (confirmed, declared or assisted), so a tracked click is never blended with a customer's memory. It also explains why Google AI Overviews clicks cannot be separated in Search Console. No SaaS, no tracking script, and no promise of more traffic: it shows what is already happening. https://normbrytande.gumroad.com/l/ai-referral-ledger

Is ChatGPT citing your site? A 30 minute way to check (and what makes a page citable)

Robin — Mon, 12 Oct 2026 08:49:21 +0000

If you ship a small SaaS, you can treat "does ChatGPT cite us?" like any other measurement problem: fixed inputs, repeated runs, a simple log, and one change at a time. Here is what the October 2026 data says, a method, and the page level changes that make a page easier to cite.

The data, briefly

CueScout (500 answers, 4,709 cited links, 50 software buying questions, 1 and 2 Oct): ChatGPT averaged 4.7 distinct pages and 3.7 searches per answer; 63% of cited pages were on the site of a brand the answer named; zero Reddit, YouTube or Wikipedia citations in 100 ChatGPT answers. Perplexity kept 73% of cited sites day to day vs 23% for ChatGPT. Vendor sells citation tracking. https://cuescout.com/blog/which-sources-chatgpt-gemini-perplexity-cite
TheToolsVerse (40 questions, ChatGPT, Gemini, AI Mode, 9 Oct): of 600 distinct cited pages, 556 (92.7%) were cited by one engine only and 2 (0.33%) by all three: a vendor pricing page and a vendor help article answering an "is it free" fact. Semrush affiliate promo content. https://thetoolsverse.com/blog/ai-citation-overlap-chatgpt-gemini-ai-mode
Kingy AI Citation Index (1,200 questions asked twice via API, 7,200 answers): cross provider cited domain Jaccard about 0.06; same provider two run overlap 0.664 (OpenAI), 0.515 (Claude), 0.319 (Gemini); native citations in 71.6%, 53.7% and 25.2% of answers. Funded by Kingy.ai; API may differ from consumer apps. https://kingy.ai/research/kaci-wave-one-ai-citation-study/
Lead Rescue (7,409 citations, 606 answers, 39 buyer questions, 8 brands): the brand asked about got 2% to 4% of citations; other companies' blogs and guides took 34% to 55%; ChatGPT pulled 24% from help docs, research, government and Wikipedia. Vendor sells citation tracking. https://leadrescue.app/resources/research/page-types-ai-engines-cite
Is My Brand in AI (50 price questions, 5 engines): median share of answer figures found on a cited page was 57% for ChatGPT, 82% AI Overviews, 71% Gemini, 92% Perplexity, 91% Claude API. Founder research for a visibility product. https://ismybrandinai.com/do-the-numbers-in-ai-answers-have-sources

Takeaway: engines are independent systems with low overlap and high run to run variance. Measure per engine, repeat, and give them a plain owned page to quote.

The method (30 minutes)

Freeze 3 to 5 buyer questions. Store them in a file. Exact strings, never edited.
Fresh chat per engine. New session, memory off where possible, one paste per question, no follow ups.
Log you / competitor / nobody, one cell per question per engine, plus a boolean for "answer had any citations". A CSV is enough:

date,run,engine,question_id,outcome,had_citations,cited_urls
2026-10-12,1,chatgpt,q1,competitor,true,"https://rival.example/pricing"
2026-10-12,1,perplexity,q1,you,true,"https://yoursite.example/pricing"
2026-10-12,1,gemini,q1,nobody,false,""

Rerun twice before calling a move. Given same provider overlap of 0.319 to 0.664 in Kingy's runs, one sample is noise. Require a pattern across at least three runs on different days.
Verify one number. For one price answer, open each cited URL and mark each figure found or not found.

What makes a page citable

None of this guarantees a citation. These are the properties shared by the pages the studies above saw cited: vendor pricing pages, help articles, and narrow fact pages.

Plain product, price and buyer statements. Put a literal sentence near the top: "Acme Invoice is an invoicing tool for freelance designers. Plans start at $X per month. The free plan covers up to 3 clients." Render it as HTML text, not inside an image, a canvas, or a client side widget that only appears after JavaScript runs. If a fetcher cannot read the price in the initial HTML, it cannot quote it.

One question per FAQ entry, answered in the first sentence. "Is Acme free for teams under 5? Yes, up to 3 clients." Then the detail. Narrow "is it free" facts were the only pages all three engines cited in the TheToolsVerse sample.

Schema basics. Structured data does not force a citation, but it makes facts unambiguous to parsers. Keep it consistent with the visible text:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "name": "Acme Invoice",
  "applicationCategory": "BusinessApplication",
  "offers": { "@type": "Offer", "price": "12.00", "priceCurrency": "EUR" }
}
</script>
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [{
    "@type": "Question",
    "name": "Is Acme Invoice free for small teams?",
    "acceptedAnswer": { "@type": "Answer", "text": "Yes. The free plan covers up to 3 clients." }
  }]
}
</script>

Crawlable and dated. Server rendered or static HTML, a stable URL, a visible "last updated" date, and nothing blocking the page in robots.txt that you actually want read.

Help docs count. Lead Rescue saw ChatGPT draw 24% of citations from help docs and similar sources. A clear docs page answering a buyer question can do more than another homepage rewrite. Lead Rescue also found other companies' guides took the largest share, so a useful guest guide elsewhere is a reasonable second move.

Ship one page, then rerun your frozen questions over two weeks and compare the log.

Free sheet

I made a free 5-prompt sheet for this: https://normbrytande.gumroad.com/l/quwzbl (and a 30-day version, The Money Prompt Lab, https://normbrytande.gumroad.com/l/wfkslz). Disclosure: I made both.