DEV Community

AURA GEO
AURA GEO

Posted on

Is That Really GPTBot? How to Find and Verify AI Crawlers in Your Server Logs

#seo #webdev #ai #devops

You've allowed AI search bots in robots.txt. Great. But are they actually reaching your pages, and are the requests in your logs really from them?

The User-Agent header is just a string. Anyone can send GPTBot in a curl request. So "I saw GPTBot in my logs" proves very little until you check two things: what status code the bot got and whether the IP really belongs to the vendor.

This guide shows how to do both with standard tools: grep, awk and a short Python script.

1. Know which bots you're looking for

Major AI vendors now separate bots by purpose. This matters, because blocking the wrong one can quietly remove you from AI search answers. According to the vendors' own documentation:

Vendor Bot What it's for
OpenAI OAI-SearchBot Surfacing sites in ChatGPT search results
OpenAI GPTBot Crawling content that may be used for model training
OpenAI ChatGPT-User Visits triggered by a user's request in ChatGPT
Perplexity PerplexityBot Surfacing and linking sites in Perplexity search
Perplexity Perplexity-User Fetches triggered by a user's question
Anthropic Claude-SearchBot Improving Claude's search results
Anthropic ClaudeBot Collecting content that may be used for training
Anthropic Claude-User Fetches when a user asks Claude a question

Two practical notes from the docs:

  • OpenAI says sites opted out of OAI-SearchBot won't be shown in ChatGPT search answers (except as navigational links), and that a robots.txt change can take about 24 hours to take effect for search. Perplexity also mentions up to 24 hours.
  • User-triggered fetchers (ChatGPT-User, Perplexity-User) are documented as acting on behalf of a user, so robots.txt rules may not apply to them in the same way.

Sources: OpenAI crawlers, Perplexity crawlers, Anthropic crawlers.

2. Find AI bot hits in your access log

The examples assume the standard nginx/Apache "combined" log format, where the client IP is the first field, the status code is the ninth, and the user agent is the sixth field when you split on double quotes.

Count requests per bot:

grep -oE 'OAI-SearchBot|GPTBot|ChatGPT-User|PerplexityBot|Perplexity-User|Claude-SearchBot|ClaudeBot|Claude-User' \
  /var/log/nginx/access.log | sort | uniq -c | sort -rn
Enter fullscreen mode Exit fullscreen mode

See which pages a specific bot requested:

grep 'OAI-SearchBot' /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
Enter fullscreen mode Exit fullscreen mode

If your logs are rotated and gzipped, use zgrep on access.log.*.gz the same way.

3. Check the status codes — this is where problems hide

A bot that "visits" but receives 403 or 429 isn't reading your content. This is very common on sites behind a CDN or WAF with aggressive bot protection.

grep -E 'OAI-SearchBot|PerplexityBot|Claude-SearchBot' /var/log/nginx/access.log \
  | awk '{print $9}' | sort | uniq -c | sort -rn
Enter fullscreen mode Exit fullscreen mode

What to look for:

  • 200 on real content pages — good.
  • 301/302 — fine if they resolve to a 200; check that redirects don't loop or land on a login page.
  • 403 — usually a firewall or bot-protection rule. Perplexity's docs, for example, describe allowing their bots in Cloudflare or AWS WAF by combining a user-agent match with their published IP ranges.
  • 429 — rate limiting. Bots may back off and come back less often.
  • 5xx — your server failed; the bot got nothing useful.

Important: if a CDN blocks a request at the edge, it may never reach your origin server logs at all. In that case, check the CDN's own firewall/security events too.

4. Verify the IP, not just the user agent

OpenAI and Perplexity publish JSON files with the IP ranges their bots use:

  • https://openai.com/searchbot.json
  • https://openai.com/gptbot.json
  • https://openai.com/chatgpt-user.json
  • https://www.perplexity.ai/perplexitybot.json
  • https://www.perplexity.ai/perplexity-user.json

Each file contains a prefixes list with ipv4Prefix or ipv6Prefix entries. Here's a small script that checks IPs from your log against them:

#!/usr/bin/env python3
"""verify_ai_bots.py — check which logged AI-bot IPs match official ranges."""
import ipaddress, json, re, sys, urllib.request

SOURCES = {
    "OAI-SearchBot": "https://openai.com/searchbot.json",
    "GPTBot": "https://openai.com/gptbot.json",
    "ChatGPT-User": "https://openai.com/chatgpt-user.json",
    "PerplexityBot": "https://www.perplexity.ai/perplexitybot.json",
    "Perplexity-User": "https://www.perplexity.ai/perplexity-user.json",
}

def load_ranges(url):
    req = urllib.request.Request(url, headers={"User-Agent": "range-check"})
    data = json.load(urllib.request.urlopen(req, timeout=20))
    nets = []
    for p in data.get("prefixes", []):
        cidr = p.get("ipv4Prefix") or p.get("ipv6Prefix")
        if cidr:
            nets.append(ipaddress.ip_network(cidr))
    return nets

ranges = {bot: load_ranges(url) for bot, url in SOURCES.items()}
seen = {}

with open(sys.argv[1], errors="replace") as log:
    for line in log:
        for bot in ranges:
            if re.search(rf"\b{re.escape(bot)}\b", line):
                ip = line.split()[0]
                seen.setdefault((bot, ip), 0)
                seen[(bot, ip)] += 1

for (bot, ip), hits in sorted(seen.items()):
    addr = ipaddress.ip_address(ip)
    ok = any(addr in net for net in ranges[bot])
    print(f"{'OK  ' if ok else 'FAKE?'} {bot:16} {ip:40} {hits} hits")
Enter fullscreen mode Exit fullscreen mode

Run it:

python3 verify_ai_bots.py /var/log/nginx/access.log
Enter fullscreen mode Exit fullscreen mode

Anything marked FAKE? claims to be an AI bot but comes from an IP outside the published ranges. Don't panic: the ranges change over time, so re-download them before drawing conclusions. But if a "GPTBot" is hammering your site from a random hosting provider, it's very likely not OpenAI.

Note: if your server sits behind a reverse proxy or CDN, the first log field may be the proxy's IP. Make sure you log the real client IP (for example via X-Forwarded-For / CF-Connecting-IP with a trusted-proxy config).

5. For Googlebot and Bingbot: reverse DNS

Google and Bing also publish IP lists (googlebot.json, bingbot.json), and both support the classic reverse-then-forward DNS check:

host 66.249.66.1
# → ...googlebot.com
host crawl-66-249-66-1.googlebot.com
# → must resolve back to 66.249.66.1
Enter fullscreen mode Exit fullscreen mode

If the reverse lookup doesn't end in a Google-owned domain (such as googlebot.com or google.com), or the forward lookup doesn't return the same IP, it isn't Googlebot. The same idea works for Bingbot with search.msn.com.

6. What a verified visit does — and doesn't — prove

A verified OAI-SearchBot hit with a 200 tells you one thing: the bot could fetch that page. It does not prove that:

  • the page is indexed in that assistant's search,
  • the page will be cited in an answer,
  • anyone will click through.

Those are separate stages, and each needs its own evidence: a saved answer that links to your page, and a real referral visit in your analytics. Treat bot logs as the first link in that chain, not the finish line.

Quick checklist

  • [ ] List AI bots in your logs and count hits per bot
  • [ ] Check status codes for search bots — no 403/429 on public pages
  • [ ] Check CDN/WAF events for edge blocks that never reach your origin
  • [ ] Verify IPs against the official JSON ranges (re-download them regularly)
  • [ ] Use reverse DNS for Googlebot and Bingbot
  • [ ] Don't treat a crawl as a citation

If you'd rather not script this by hand, AURA GEO runs these accessibility checks for your site, keeps a crawl history for AI bots and shows real referral visits from AI assistants — and your own agent can pick up the fixes over MCP. There's also an English write-up on why a bot visit isn't a citation.

Top comments (2)

Collapse
 
klopsy91 profile image
klopsy91 •

Good point about the status codes being where problems hide. One thing I'd add: if the CDN blocks at the edge, the origin logs show nothing, so it's worth also checking the CDN's own bot/firewall logs for 403s on OAI-SearchBot and PerplexityBot. I've seen sites that looked fine in robots.txt but were silently challenging those bots at the WAF. Do you also compare the 200 responses against what a plain curl without JavaScript returns, to make sure the content is actually there?

Collapse
 
aurageo profile image
AURA GEO •

Thanks, great point about the CDN logs. Yes, comparing against a plain curl (no JavaScript) is a good extra check: if the 200 response is missing the main content, bots that don't render JS will see the same empty page. A quick curl with the bot's user agent string is a handy sanity test, though a WAF may still treat it differently than the real bot IP.