DEV Community: Ali Baizhanov

How to see which AI crawlers read your website (5 ways, with copy-paste commands)

Ali Baizhanov — Tue, 13 Oct 2026 09:59:02 +0000

Originally published on pagelens.io.

AI crawlers read websites all day, and almost no site owner can see it. The reason is simple: analytics tools count visitors with JavaScript that runs in a browser, and AI crawlers download the HTML without running any JavaScript. To see them you have to look where the request arrives: your server or your CDN.

Here are five ways to do that, starting with the ones that cost nothing.

First: which bots to look for

There are three kinds, and they mean different things:

Kind	Examples	What a request means
Training crawlers	GPTBot, ClaudeBot, CCBot	The page was collected for training a model
AI search crawlers	OAI-SearchBot, Claude-SearchBot, PerplexityBot	The page went into the index an assistant searches
Fetches for a person	ChatGPT-User, Claude-User, Perplexity-User	Someone asked an assistant about this right now, and it opened your page

The third kind is the one to watch. It is the closest thing to "my page was used in an answer".

1. Server access logs (free, if you run the server)

Nginx and Apache already write every request to a log. Count requests per AI bot:

grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn

See which pages one of them asked for (the path is the 7th field in the default log format):

grep "ChatGPT-User" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

And the requests that ended in a 404, which are pages a bot expected to find and didn't (the status is the 9th field):

grep -E "GPTBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log | awk '$9 == 404 {print $7}' | sort | uniq -c | sort -rn | head

Good for: a one-off look on a VPS. Limits: logs rotate after days or weeks, a CDN cache in front of the server hides requests from it, and you get a number, not a trend.

2. Your CDN's dashboard (free, if your site runs through one)

A CDN sees every request before your server does, including the ones it answers from cache. Cloudflare has an AI crawler section in its dashboard that lists AI crawlers requesting your site and lets you allow or block each. Other CDNs and hosts (Vercel, Netlify, Fastly) show requests in their logs, where you can filter by user agent.

Good for: a quick answer to "which AI companies crawl me, and how much". Limits: usually no view per page over time, and nothing about what happened next (did anyone visit, did they buy).

3. A few lines of middleware (free, for Next.js and similar)

If your framework lets you run code on every request, log the bots there. In Next.js:

// middleware.ts (proxy.ts in Next.js 16)
import { NextResponse, type NextRequest } from 'next/server'

const AI = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User/i

export function middleware(req: NextRequest) {
  const bot = req.headers.get('user-agent')?.match(AI)?.[0]
  if (bot) console.log(JSON.stringify({ bot, path: req.nextUrl.pathname }))
  return NextResponse.next()
}

export const config = { matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'] }

The lines land in your host's function logs. The same idea works in Express, a Cloudflare Worker or any server framework.

Limits: you still have to store, count and chart them, and middleware runs before the page, so it doesn't know the response status.

4. WordPress

WordPress runs PHP on every uncached request, so a short snippet in a small plugin or your child theme can log AI bots:

add_action('template_redirect', function () {
    $ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
    if (preg_match('/GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User/i', $ua, $m)) {
        error_log('AI bot ' . $m[0] . ' ' . ($_SERVER['REQUEST_URI'] ?? '/'));
    }
});

The catch: page caches (WP Rocket, LiteSpeed, a CDN) serve cached pages without running PHP, so cached pages go uncounted. Exclude bot user agents from the cache, or count at the CDN instead.

5. Analytics that reads bots from your server (automatic)

The four ways above give you raw requests. Turning them into something you can act on means storing them, grouping them by page and by company, and joining them with what people did afterwards. I built this into PageLens: a small snippet on your server reports bot requests, and the dashboard shows which pages AI crawlers read, which ones assistants opened to answer people, the 404s they hit, and the visits and revenue that came from AI assistants.

Check that the bots are real

Anyone can send a request that says GPTBot. OpenAI, Anthropic and Perplexity publish the IP ranges their bots use, so a request with a bot's name from another address is an impostor. When I checked 740 top websites, 9% turned away requests carrying AI bot names even though their robots.txt allowed those bots.

What to do with the numbers

Read a lot, never opened for an answer: AI knows the page exists and doesn't find it useful for questions. Put the question in the heading and the answer in the first two sentences.
Opened for answers, no visits: you are being used as a source without being clicked. Check how the page describes you and whether it gives a reason to click through.
Important page never read: link to it from your home page and sitemap, and check robots.txt and your CDN.
Bots hitting 404s: they expect a page there, often /pricing, /docs or /llms.txt. Create it or redirect.

The field numbers in the commands are for the default "combined" log format of Nginx and Apache. If your format differs, adjust them.

What do you use to watch AI crawlers on your sites?

We checked 740 of the world's top websites for AI bots. Most that block GPTBot also block ChatGPT search

Ali Baizhanov — Mon, 12 Oct 2026 08:27:34 +0000

Originally published on pagelens.io.

Most advice about AI search starts with "check your robots.txt". I wanted to know what the biggest sites actually do, so I ran an AI visibility check on the top 1,500 domains of the Tranco list (list 4672X), a research ranking of the most visited domains, on October 12, 2026.

Many of those domains are APIs and CDNs with no home page (gstatic.com, akamai.net and so on). 740 served a home page to a normal browser request, and every number below is out of those 740 websites.

The short version

15% block GPTBot in robots.txt, and about as many block ClaudeBot (14%) and Google-Extended (13%).
13% block at least one agent that answers people: OAI-SearchBot, ChatGPT-User, Perplexity's bots or Claude-User.
Of the 109 sites that block GPTBot, 69 (63%) also block ChatGPT's search or browsing agent. Only 40 block training and keep ChatGPT search open.
9% turn away requests that carry an AI bot's name at the server, even though their robots.txt allows those bots.
About 14% publish an llms.txt; 45% have structured data on the home page; 38% have no sitemap I could find.

1. robots.txt: training bots are blocked more than answer bots

Share of the 740 websites whose robots.txt keeps each agent off the home page (in practice, off the site):

Agent	What it does	Blocked
CCBot	Common Crawl, an open dataset many AI labs train on	16.8%
Bytespider	ByteDance training crawler	15.7%
GPTBot	OpenAI training crawler	14.7%
ClaudeBot	Anthropic training crawler	13.6%
Google-Extended	Gemini training (a robots.txt token, not a separate crawler)	13.2%
meta-externalagent	Meta training crawler	12.6%
PerplexityBot	Perplexity search index	10.9%
ChatGPT-User	ChatGPT opening a page for a person	9.7%
Applebot-Extended	Apple Intelligence training	9.3%
Perplexity-User	Perplexity opening a page for a person	8.4%
Claude-User	Claude opening a page for a person	8.0%
OAI-SearchBot	ChatGPT search index	7.8%

Most of these are deliberate: 92 of the 109 GPTBot blocks name GPTBot explicitly, the rest come from a rule for all bots.

Each agent has a page in this AI bots directory with its user agent, robots.txt rules and published IP ranges.

2. Blocking GPTBot usually blocks ChatGPT search too

GPTBot only collects training data. Whether a site appears in ChatGPT search depends on OAI-SearchBot, and ChatGPT-User is the agent that opens a page while ChatGPT answers someone. OpenAI documents them as separate agents. One nuance: OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person starts those requests, so a robots.txt block of it is a stated preference more than a guarantee.

Of the 109 websites that block GPTBot:

69 also block OAI-SearchBot or ChatGPT-User. Among them are cnn.com, bbc.com, reddit.com, imdb.com, espn.com, nature.com and Amazon's stores. They opt out of ChatGPT as a whole.
40 block only training and leave ChatGPT search and browsing open, including forbes.com, bloomberg.com, airbnb.com, figma.com, ted.com and nintendo.com.

Both are legitimate choices. The second one is what most businesses want: stay out of training, stay in the answers. If you copied a "block AI bots" robots.txt from a blog post, check which group you're in.

3. robots.txt isn't the whole story

For each site I also requested the home page with each bot's name in the user agent (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot) and compared the answer with a normal browser request. On 130 sites, at least one bot-named request was turned away (a 403, a challenge page or no answer) although robots.txt allows that bot.

I then re-tested those 130 sites with a fourth request: a generic, made-up crawler name. That splits them into groups:

69 (9.3% of all sites) let the browser and the generic crawler through, but turn away requests named after AI bots. 37 of them turn away ChatGPT-User, the agent that opens pages to answer people.
37 (5%) turn away any bot-like request, AI or not.
19 did not repeat on the second test, and 5 blocked the browser too.

One caution: these requests don't come from OpenAI's or Anthropic's servers. Some CDNs block requests that claim to be GPTBot but don't come from the IP ranges OpenAI publishes, and let the real GPTBot in. From the outside, "blocks AI bots" and "blocks impostors" look the same, which is why I don't name sites in this group. If you're one of them, your CDN's bot settings (in Cloudflare: Security → Bots) will tell you which it is.

The useful lesson for a site owner: robots.txt can say "allowed" while the server says no, and nothing in robots.txt will show it.

4. Can AI read the page once it gets it?

AI crawlers don't run JavaScript. On 22% of home pages, fewer than 120 words were readable in the HTML before scripts run. This is an upper bound: some sites send bots a short interstitial page instead of the real one. But for a single-page app, an empty HTML shell is what every AI crawler sees.

Signal	Share of the 740 sites
Has a robots.txt	76%
Has a sitemap (in robots.txt or at /sitemap.xml)	62%
Has structured data (JSON-LD) on the home page	45%
Missing a title or meta description	21%
Publishes an llms.txt (Markdown with a title)	14%

llms.txt is still a minority practice even among the largest sites: about one in seven. Several more sites answered /llms.txt with something that isn't one (an IP address, a bare "OK", an error page), which a naive check would count.

What to do on your own site

Decide training and answers separately. Blocking GPTBot and ClaudeBot keeps you out of training; to stay in ChatGPT and Claude answers, allow OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User.
Check the server, not only the file. CDN bot settings and firewalls override robots.txt.
Make sure the text is in the HTML. Server rendering or prerendering for the pages you want quoted.
Then measure what AI actually does. Server logs show which pages AI crawlers read and which ones assistants open to answer people.

If you want to run the same check on your site, I built it as a free tool (no sign-up): pagelens.io/tools/ai-crawler-checker.

Method and limits

Domains: top 1,500 of Tranco list 4672X. A domain counts as a website if a normal browser request to its home page got a page (2xx or 3xx, no challenge). 740 did.
robots.txt rules were evaluated for the home page path with the most specific matching user-agent group, as crawlers do.
Bot requests used each bot's name and said they came from the checker. Run from Central Asia on October 12, 2026; results can differ by location and change at any time.
These are shares, not judgements: blocking AI bots is a valid choice.

Happy to answer questions about the data or the method in the comments.