Cloudflare Web Analytics told me chatgpt.com was one of my largest referrers. That is the sum total of what it could tell me, because Web Analytics is a client-side beacon: it fires from a browser running JavaScript. Every crawler on earth is invisible to it by construction.
So when I shipped a page specifically for AI assistants to read, I had no way to answer the only question that mattered — did any of them come?
I added about a hundred lines of server-side counting to find out. The result was not what I expected, and it made me throw away part of the work I had just done.
Three hats, not one
The naive version of this is a counter that increments on "is this a bot". That number is useless, and it took me one iteration to see why: an AI agent hits your site wearing one of three completely different hats.
Index. A crawler walking on its own schedule, building a corpus. OAI-SearchBot, PerplexityBot, Claude-SearchBot. This says you are reachable.
On-behalf. A person asked a question a second ago, and the assistant is fetching your page right now to answer it. ChatGPT-User, Perplexity-User, Claude-User. This says you are being cited. This is the number.
Training. Corpus collection for model training. GPTBot, ClaudeBot, Google-Extended. Neither of the above.
All three of OpenAI's agents carry the string "OpenAI" in the wild, and a matcher that collapses them produces a number that looks like data and answers nothing. Order matters when you match, too — test chatgpt-user before gptbot, or a naive substring search folds them together.
// UA substring -> [canonical name, hat]. First hit wins, so the more specific
// spelling comes first.
const AGENTS = [
["oai-searchbot", ["OAI-SearchBot", "index"]],
["chatgpt-user", ["ChatGPT-User", "on-behalf"]],
["gptbot", ["GPTBot", "training"]],
["perplexity-user", ["Perplexity-User", "on-behalf"]],
["perplexitybot", ["PerplexityBot", "index"]],
["claude-user", ["Claude-User", "on-behalf"]],
["claude-searchbot", ["Claude-SearchBot", "index"]],
["claudebot", ["ClaudeBot", "training"]],
["googlebot", ["Googlebot", "index"]],
["bingbot", ["Bingbot", "index"]],
];
function classify(ua) {
const s = String(ua || "").toLowerCase();
for (const [needle, hit] of AGENTS) {
if (s.includes(needle)) return { name: hit[0], hat: hit[1] };
}
return null; // a person, or something that does not say
}
There is a fourth number worth having, and it is not a crawler at all. ChatGPT stamps the links it renders with utm_source=chatgpt.com. Counting those gives you the click-through: a human who saw a citation and decided it was worth opening.
Read together, the four are a funnel. Index arrives first. On-behalf follows once your page is actually retrievable. Clicks follow once a person decides the citation was worth a tap.
Keeping it cheap and keeping it private
Two constraints shaped the implementation more than anything else.
It is fed by strangers. The traffic this measures is exactly the traffic you do not control. A crawler walking my archive is a thousand page requests. So: aggregate in memory, keyed by day and path, and persist on a debounced flush rather than per request. If you mirror to a paid key-value store, mirror the accumulated day — one write every thirty seconds while anything is arriving, none when it is not. A crawler flood and an idle afternoon then cost exactly the same. Bound the number of distinct paths per agent per day too, or a single deep crawl grows one key per URL forever.
It should not become surveillance. There is no reason for this instrument to hold anything about a person. Mine records which known agent matched, which path, and the date. No IP, no account, no session, no user-agent string kept verbatim, nothing from the query string except that one attribution parameter. Counts roll off after sixty days. That is enough to answer "did OpenAI fetch this page yet" and not enough to answer anything about anyone.
If you have a privacy policy that says page views are counted by your analytics provider and stops there — mine did — that sentence stopped being the whole truth the moment this landed. Update it.
Two bugs worth stealing
The shutdown flush that never ran. My server registered signal handlers at module load: process.on(sig, () => process.exit(0)). My counter registered its own flush handler later, from inside the listen() callback. Node runs listeners in registration order, the first one calls process.exit(0), and the second never fires. A deploy is a SIGTERM, and a deploy is precisely when someone is watching the last few hours of data. The fix is not a second handler — it is to flush inside the one that already owns shutdown.
Load-or-reset. open() read a file and, if the file was missing, left whatever was already in memory. A second call silently inherited the first one's counts. "Load whatever survived" has to mean empty when nothing did, or the persisted file stops being the record it claims to be. A test caught this, not a code review.
The finding
Half an hour after it shipped, this is what the counter said:
THE FUNNEL
reachable (index bots) 9
cited now (on-behalf) 15 <- a person was answered from us
training 0
BY AGENT
ChatGPT-User on-behalf 15
/research/level-hold-rates 1
/research/what-monthly-expiration... 1
/learn 1
/about 1
/mcp 1
/ 1
/sessions 1
... five session pages, four research posts
Googlebot index 9
... eight archive pages and /pricing
CLICK-THROUGHS
(nothing yet)
Three things jumped out.
ChatGPT-User outnumbered Googlebot. Fifteen on-behalf fetches to nine index crawls, on a site with a rounding error's worth of search traffic. People are asking assistants questions and the assistants are reading my pages to answer. Not skimming a title — pulling research posts, the explainer index, the docs page.
The page I built for them was never fetched. I had shipped, that same morning, a URL that answers the single most common question about this subject in plain server-rendered HTML. It appears nowhere in that list. Neither does OAI-SearchBot, which had not visited at all.
Nobody clicked. Fifteen reads, zero click-throughs.
What that actually means
The middle finding is the one that changed my plans, and it took a minute to understand.
The assistant fetched / and /sessions. My new page was linked from the navigation of both, and named in a sentence on one of them. It did not go.
In search mode, an assistant opens what its index returned. It does not browse — it does not follow links out of a page it just read. So a page that is hours old and indexed nowhere is unreachable no matter how prominently you link it internally. Every internal link I had added that morning was, for this purpose, worthless.
And it gets more pointed. / was fetched — and / was a JavaScript application shell whose HTML carried no numbers at all. The most-fetched URL on the site was answering the question with nothing.
So I moved the answer to where the fetch already lands: the current reading now renders into the shell's HTML as a <noscript> block. Not a hidden div served only to crawlers — the same bytes for everyone, and a genuine fallback, because without JavaScript that page really is blank. That is the line between a fallback and cloaking, and it is worth staying on the right side of it.
What I would tell you to do
Count the three hats separately. One "bot hits" number will not tell you whether you are indexed or whether you are being cited, and those need different responses.
Check whether your most-fetched URL actually answers anything. If it is a JS app shell, an assistant is spending a fetch on your site and leaving with nothing. That is the cheapest fix available and I would not have found it without the data.
Do not assume internal links help here. They help humans and they help traditional crawlers. They appear not to help an assistant that only opens search results.
Watch the click-through row. Fifteen reads and zero clicks is one afternoon on one small site and I would not generalise from it. But it is the number that separates "we are in the answer" from "we get the visit", and if you are not counting it you will not notice which one you have.
The honest caveat on all of this: it is roughly thirty minutes of data from a single site in a niche, and some of those fetches were me testing. It is not a study. It is an instrument that took an afternoon to build and immediately told me something I had got wrong — which is the only real argument for building it.
The site is gex.live, which rebuilds SPX dealer positioning from the options tape. If you want the flavour of what the assistants were pulling: how often dealer-gamma levels hold, measured across a thousand sessions, and what monthly expiration does to the book. Most of what is published there is a null result, which is its own kind of fun.
Top comments (1)
crazy how much of a difference the referrer data makes vs just looking at the user agent strings lol