A six-step check for ChatGPT and other AI-assistant traffic, built from the numbers founders shared under my last post.
A founder who commented on my last post had a funnel that looked broken. Over seven days their analytics showed 206 sign-in starts and 0 completed signups. Numbers like that send you straight into the auth code.
The code was fine. Of 205 clicks on the Google sign-in button in four days, about 190 came from Germany. Each one was a different visitor. Many reported browser and OS pairs that don't exist in practice, like Safari on Linux. Once the founder excluded that traffic, 7-day visitors dropped from 358 to roughly 85, and sign-in starts dropped from 206 to 1. That single remaining start was their own test.
Nothing in the funnel was broken. The visitors weren't real, and any conversion rate calculated that week would have had a fake denominator.
My last post argued that AI referrals should be judged by what they turn into. Several readers answered with their own data, which was more useful than anything I wrote. Here it is as a method you can set up in an afternoon and leave running for a month.
Step 1: Remove traffic that isn't people
Do this before you look at any source. If junk is in your totals, every rate you compute inherits it.
What gave it away in the case above:
- One country suddenly owning one event. 190 of 205 clicks from a single country is a script, not a market.
- Every session is a new visitor, with no returns.
- Browser and OS pairs that don't ship. Apple doesn't make Safari for Linux, so "Safari on Linux" in your data is a user-agent string someone typed.
Exclude those, note what you excluded and why, and keep the before and after numbers.
AI crawlers like GPTBot or PerplexityBot are a separate issue. They mostly don't run your analytics script, so count them in server logs as crawl, never as visitors.
Step 2: Tag AI sources by referrer and by utm_source
Another founder shared a 30-day split from their product. AI-assistant visitors signed up at 33.7%. Google organic visitors signed up at 24.0%. That's a real gap, and the kind of number I hoped readers would find.
The detail I'd pay more attention to: 877 of their 936 ChatGPT visits arrived with an empty referrer. They were only identifiable as ChatGPT because the links carried utm_source=chatgpt.com. Group by referrer alone and ChatGPT shrinks to 59 visits at most, with the other 877 sitting in Direct.
So build your AI segment on both fields. In GA4, Plausible, PostHog or Fathom, save one filter where source, referrer or utm_source contains chatgpt, perplexity, gemini, copilot or claude. Then open five of those sessions and check that they look like people: a landing page, some time on site, an event. A value in the source column is a hint. It isn't proof.
One gap you can't close here. Google counts clicks from AI Overviews and AI Mode inside the normal "Web" search type in Search Console, so they can't be pulled out. Log them as Google.
Step 3: Pick one downstream action
A third commenter gave me the cleanest test I've seen. Choose one action a visitor can take in their first session, such as an email capture, a preview open or a checkout start. Only one. Then watch it next to your AI referral count.
If AI referrals rise and that action stays flat, your answer is no. Those visitors aren't doing anything yet, however good the traffic chart looks.
The same commenter added a check I now run first: look at which page ChatGPT actually links to. It's often not the page you would pick. If the landing page doesn't make the next step obvious, a flat action count may be telling you about the page rather than the visitor.
One action, not five: with small numbers, five metrics give you five chances to mistake noise for a win.
Step 4: Compare against Google organic in the same window
A conversion rate on its own says very little. 33.7% looks strong next to 24.0%. Next to 45% it would look ordinary.
Google organic is the right baseline because it's the closest cousin of AI traffic: someone asked a question and clicked a result. Same date range, same action, same junk filters on both groups.
Step 5: Split by device before you call it an AI effect
The founder with the 33.7% figure noticed that most of their ChatGPT visits came from phones. That matters. Phone and desktop visitors convert differently on most sites, and the direction depends on the product. If your Google traffic is mostly desktop and your ChatGPT traffic is mostly mobile, part of the gap might be the device and not the assistant.
Cut the comparison into four cells: AI on phone, AI on desktop, Google on phone, Google on desktop. If AI still wins inside each device group, you have something worth acting on. If the gap disappears, you've saved yourself from rewriting your positioning around a phone effect.
Step 6: Run it for 30 days
A week tells you very little. The 206/0 case shows how one bad week can take over a dashboard. Thirty days gives you enough rows to see whether the action moves with referrals, and time to change one thing (usually the landing page).
While you wait, add a "How did you first hear about us?" field to signup or checkout, with ChatGPT, Perplexity, Gemini, Microsoft Copilot and Claude as separate options. Some AI visitors arrive with no referrer and no UTM at all, and asking is the only way to find them. Keep those answers in their own column. A customer saying "I found you on Perplexity" is real evidence, but it is weaker than a tracked session, and mixing the two inflates your total.
My view after these comments: most small-site claims that AI traffic "converts amazingly" or "doesn't convert" were never checked. Both the fake-traffic case and the empty-referrer case would have produced a confident, wrong answer from uncleaned data.
The kit I use for the 30 days
I packaged my version of this routine as The AI Referral Ledger. It costs $19 and it's files only: no SaaS, no tracking script, no dashboard.
You get an 11-page PDF guide, a ledger CSV and a markdown file with the copy-paste form field and a week-by-week checklist. The guide covers the source, referrer and utm_source filter, the five-session sanity check, why Search Console can't separate AI Overviews clicks, and what the vendor-documented bots say they do. Every AI-attributed row gets one label: confirmed (your analytics shows the AI session and the action), declared (the customer told you) or assisted (AI was part of the journey but not the last step you can see). The sheet has columns for landing page, the question the customer typed into the assistant, the action and its value, so you can see which pages lead anywhere. Week 1 takes about an hour, then about 30 minutes a week.
What it doesn't do: it won't spot fake traffic for you, and it doesn't run the device split. Those are filters you set in your own analytics tool, as described above. It won't increase your AI traffic either. It records what is already happening, and your numbers may be small.
Get The AI Referral Ledger ($19)
Not sure yet whether AI assistants mention you at all? Start with The 5-Prompt Citation Snapshot. It's free: a starter log for five fixed prompts across engines, with separate cells for "named" and "cited". If nobody names you, there's no referral traffic to measure yet.
If you've run a split like the ones above, I'd like to see it. Did AI visitors still beat Google once you split by device?
Top comments (1)
Great checklist. Splitting by traffic source before panicking about the funnel is such a good habit.