How to see which AI crawlers read your website (5 ways, with copy-paste commands)

Originally published on pagelens.io.

AI crawlers read websites all day, and almost no site owner can see it. The reason is simple: analytics tools count visitors with JavaScript that runs in a browser, and AI crawlers download the HTML without running any JavaScript. To see them you have to look where the request arrives: your server or your CDN.

Here are five ways to do that, starting with the ones that cost nothing.

First: which bots to look for

There are three kinds, and they mean different things:

Kind	Examples	What a request means
Training crawlers	GPTBot, ClaudeBot, CCBot	The page was collected for training a model
AI search crawlers	OAI-SearchBot, Claude-SearchBot, PerplexityBot	The page went into the index an assistant searches
Fetches for a person	ChatGPT-User, Claude-User, Perplexity-User	Someone asked an assistant about this right now, and it opened your page

The third kind is the one to watch. It is the closest thing to "my page was used in an answer".

1. Server access logs (free, if you run the server)

Nginx and Apache already write every request to a log. Count requests per AI bot:

grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn

See which pages one of them asked for (the path is the 7th field in the default log format):

grep "ChatGPT-User" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

And the requests that ended in a 404, which are pages a bot expected to find and didn't (the status is the 9th field):

grep -E "GPTBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log | awk '$9 == 404 {print $7}' | sort | uniq -c | sort -rn | head

Good for: a one-off look on a VPS. Limits: logs rotate after days or weeks, a CDN cache in front of the server hides requests from it, and you get a number, not a trend.

2. Your CDN's dashboard (free, if your site runs through one)

A CDN sees every request before your server does, including the ones it answers from cache. Cloudflare has an AI crawler section in its dashboard that lists AI crawlers requesting your site and lets you allow or block each. Other CDNs and hosts (Vercel, Netlify, Fastly) show requests in their logs, where you can filter by user agent.

Good for: a quick answer to "which AI companies crawl me, and how much". Limits: usually no view per page over time, and nothing about what happened next (did anyone visit, did they buy).

3. A few lines of middleware (free, for Next.js and similar)

If your framework lets you run code on every request, log the bots there. In Next.js:

// middleware.ts (proxy.ts in Next.js 16)
import { NextResponse, type NextRequest } from 'next/server'

const AI = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User/i

export function middleware(req: NextRequest) {
  const bot = req.headers.get('user-agent')?.match(AI)?.[0]
  if (bot) console.log(JSON.stringify({ bot, path: req.nextUrl.pathname }))
  return NextResponse.next()
}

export const config = { matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'] }

The lines land in your host's function logs. The same idea works in Express, a Cloudflare Worker or any server framework.

Limits: you still have to store, count and chart them, and middleware runs before the page, so it doesn't know the response status.

4. WordPress

WordPress runs PHP on every uncached request, so a short snippet in a small plugin or your child theme can log AI bots:

add_action('template_redirect', function () {
    $ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
    if (preg_match('/GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User/i', $ua, $m)) {
        error_log('AI bot ' . $m[0] . ' ' . ($_SERVER['REQUEST_URI'] ?? '/'));
    }
});

The catch: page caches (WP Rocket, LiteSpeed, a CDN) serve cached pages without running PHP, so cached pages go uncounted. Exclude bot user agents from the cache, or count at the CDN instead.

5. Analytics that reads bots from your server (automatic)

The four ways above give you raw requests. Turning them into something you can act on means storing them, grouping them by page and by company, and joining them with what people did afterwards. I built this into PageLens: a small snippet on your server reports bot requests, and the dashboard shows which pages AI crawlers read, which ones assistants opened to answer people, the 404s they hit, and the visits and revenue that came from AI assistants.

Check that the bots are real

Anyone can send a request that says GPTBot. OpenAI, Anthropic and Perplexity publish the IP ranges their bots use, so a request with a bot's name from another address is an impostor. When I checked 740 top websites, 9% turned away requests carrying AI bot names even though their robots.txt allowed those bots.

What to do with the numbers

• Read a lot, never opened for an answer: AI knows the page exists and doesn't find it useful for questions. Put the question in the heading and the answer in the first two sentences.
• Opened for answers, no visits: you are being used as a source without being clicked. Check how the page describes you and whether it gives a reason to click through.
• Important page never read: link to it from your home page and sitemap, and check robots.txt and your CDN.
• Bots hitting 404s: they expect a page there, often /pricing, /docs or /llms.txt. Create it or redirect.

The field numbers in the commands are for the default "combined" log format of Nginx and Apache. If your format differs, adjust them.

What do you use to watch AI crawlers on your sites?

Comments

[Dmitry Sergeev]

We need to produce a comment as a regular developer, short, one or two sentences, fragment okay. Should start with a specific reaction or question about this video, not generic praise. Should not start with "Great video". Should be informal. Maybe ask about which AI crawlers are most common, or mention which command they used. Let's craft: "lol, the curl command for GPTBot actually works, anyone else got a 403 on the Bingbot one?" That is specific. No punctuation issues. Ensure no em dash.

[Dmitry Sergeev]

We need to produce a short comment, 1-2 sentences, casual, start with lowercase, mention specific reaction or question about video. No markdown, no quotes. Just comment text. Must be short, maybe ask about which crawlers missed? Something like "any chance these commands work on Cloudflare-protected sites?" Provide observation: "i tried the curl command but got a 403". Ensure no double hyphens. No URLs. No em dash. Use straight quotes if any. But we can avoid quotes. Provide