Originally published on LetsLaunch.
We wrote before about whether an AI
crawler can read your page at all — rendering, robots.txt mistakes, a 403 from
your own firewall. That is a one-time question. You fetch the page as
OAI-SearchBot, you look at what comes back, you fix it or you don't, and the
answer stays true until you change something.
This post is about a different question: whether these crawlers actually show
up, on an ongoing basis, once the one-time check passes. A page that was
readable to GPTBot last month tells you nothing about whether GPTBot has
fetched it since. Readability is a fact about your server. Traffic is a fact
about their behavior, and it changes without you touching anything.
What you're actually looking for
Every major AI crawler identifies itself with a distinct user-agent string.
The ones worth watching for:
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User,
Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended,
Applebot-Extended, Bingbot, Amazonbot, DuckAssistBot, Bytespider,
CCBot.
LetsLaunch's own robots.txt (src/app/robots.ts in this codebase) names
several of these explicitly rather than relying on a wildcard rule, for the
same reason you'd want to grep for them by name: a rule or a log line that
just says "bot" tells you nothing about which one.
Two of these strings are worth separating in your head before you start
counting anything, because they answer different questions.
Training crawlers and retrieval crawlers are not the same signal
We made this distinction in the readability post and it matters just as much
for monitoring: GPTBot collects training data. OAI-SearchBot and
PerplexityBot fetch pages to serve as citations in an actual answer. They
run independently, on their own schedules, for their own purposes.
If you only track GPTBot, a busy week of hits tells you your content might
end up in a future training run — and tells you nothing about whether you can
be cited in an answer today. If you want to know about citation potential, you
need to watch OAI-SearchBot, Claude-SearchBot and PerplexityBot
separately, not lump every AI-labeled agent into one counter. A dashboard that
reports "247 AI bot hits this week" without splitting training from retrieval
is reporting a number nobody can act on.
Grepping your own logs
If you run your own server or have access to raw access logs, this is a grep
away. Standard combined log format puts the request path, status code, and
user-agent on one line, so a single pipeline gets you counts per crawler:
grep -E "GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot|Claude-SearchBot" \
access.log \
| awk -F'"' '{print $6}' \
| sort \
| uniq -c \
| sort -rn
awk -F'"' splits the line on quote characters, and in combined log format
the user-agent is the sixth quoted field — the request line and the referer
are the fourth and there's no fifth field before it. sort | uniq -c turns
that into a count per exact user-agent string, which is enough to tell you
who's showing up and how often.
To see what a specific crawler is actually requesting, rather than just how
often it shows up:
grep "PerplexityBot" access.log | awk '{print $7, $9}' | sort | uniq -c | sort -rn
Field 7 is the request path and field 9 is the status code in the default
combined format (exact field numbers shift if your log adds extra fields, so
check one line by hand before trusting the count). This tells you two useful
things at once: which pages a crawler actually cares about, and whether it's
getting 200s or something else on the way in.
Run this weekly, or wire it into whatever log-shipping you already have, and
you get a trend instead of a single data point. A one-time curl check answers
"can it read this page." A weekly grep answers "does it come back."
The version with no log access required
Most sites don't sit in front of raw logs — they sit behind Cloudflare, which
is also LetsLaunch's own setup. Cloudflare's dashboard has an AI Crawlers
section, under its AI Bots reporting, that shows this without any grep at all:
total requests from AI crawlers, a breakdown of allowed versus blocked
requests, a per-crawler split naming the major players — Google, OpenAI,
Anthropic, Perplexity, ByteDance, Common Crawl, DuckDuckGo among them — and
the paths those crawlers hit most.
If your site is already on Cloudflare, this is the first place to look before
building anything yourself. It's the same information the grep gives you,
already split by crawler, already trending over time, with nothing to
instrument. The tradeoff is that it only sees what passes through Cloudflare's
edge — if you serve some paths from elsewhere, those don't show up here.
Cloudflare's own traffic data shows AI bot traffic rising sharply year over
year. We won't put a number on that, because the precise figure varies by
which report and which time window you're reading, and a specific percentage
attributed loosely is worse than no percentage at all. The direction is the
useful part: this is traffic worth watching, not a rounding error, and it is
worth setting up a recurring check rather than a one-time one.
If you just want a fast answer today
Setting up log monitoring or checking a Cloudflare dashboard is worth it if
you want a trend. If you just want to know right now whether a specific
crawler can reach a specific page, that's a narrower and faster question, and
it's what our AI crawler check does: it fetches your
page as each crawler, once, and shows you the status code and the text each
one received. It won't tell you whether ClaudeBot visited last Tuesday. It
will tell you, in the next thirty seconds, whether it could visit at all. Pair
that with our guide to making a page AI-readable if
the answer comes back wrong.
What a crawler hit does and doesn't prove
Here's the part worth being honest about, because it's the same caution we
applied to citations in the readability post. Seeing PerplexityBot in your
logs is necessary evidence. It is not sufficient evidence.
A hit tells you the crawler reached your page and (assuming a 200 and
readable HTML) received your content. It does not tell you that content was
used, indexed, or ever surfaced to a single user. You cannot infer citation
frequency from crawl frequency — a crawler can fetch a page a hundred times
and cite it never, and there's no dashboard, ours included, that closes that
gap.
What monitoring does give you is the other direction, which is just as
useful: absence of hits is real information. If OAI-SearchBot has never
touched your domain, you cannot be cited from that index, full stop — no
amount of good content changes that if the crawler was never there to read
it. Monitoring won't tell you you're winning. It will tell you, reliably,
when you've been shut out before you've done anything else.
Treat crawler-hit counts as a diagnostic, not a growth metric. They rule
things out. They don't rule things in.
Do crawler visits mean my product will be cited by ChatGPT or Perplexity?
No. A crawler hit only confirms the page was fetched and, if it returned a
readable 200, that the content was received. Whether it's later cited in an
answer is a separate step nobody can currently observe from the outside, and
we know of no reliable way to predict it from crawl frequency alone.
Do I need Cloudflare to monitor AI crawler traffic?
No, but it removes the setup work. Cloudflare's AI Crawlers analytics shows
per-crawler request counts, allowed-versus-blocked splits, and top crawled
paths for any site behind its network, with no log access needed. Without
Cloudflare, the same information is available by grepping your raw access
logs for the relevant user-agent strings.
What's the difference between monitoring GPTBot and monitoring OAI-SearchBot?
GPTBot hits reflect training-data collection. OAI-SearchBot hits reflect
retrieval for citations in ChatGPT's search answers. They're separate
crawlers with separate purposes, so tracking only GPTBot tells you nothing
about whether you can currently be cited — you have to watch the
search/retrieval agents separately to answer that question.
Top comments (0)