DEV Community: AURA GEO

Is That Really GPTBot? How to Find and Verify AI Crawlers in Your Server Logs

AURA GEO — Wed, 21 Oct 2026 16:14:49 +0000

You've allowed AI search bots in robots.txt. Great. But are they actually reaching your pages, and are the requests in your logs really from them?

The User-Agent header is just a string. Anyone can send GPTBot in a curl request. So "I saw GPTBot in my logs" proves very little until you check two things: what status code the bot got and whether the IP really belongs to the vendor.

This guide shows how to do both with standard tools: grep, awk and a short Python script.

1. Know which bots you're looking for

Major AI vendors now separate bots by purpose. This matters, because blocking the wrong one can quietly remove you from AI search answers. According to the vendors' own documentation:

Vendor	Bot	What it's for
OpenAI	`OAI-SearchBot`	Surfacing sites in ChatGPT search results
OpenAI	`GPTBot`	Crawling content that may be used for model training
OpenAI	`ChatGPT-User`	Visits triggered by a user's request in ChatGPT
Perplexity	`PerplexityBot`	Surfacing and linking sites in Perplexity search
Perplexity	`Perplexity-User`	Fetches triggered by a user's question
Anthropic	`Claude-SearchBot`	Improving Claude's search results
Anthropic	`ClaudeBot`	Collecting content that may be used for training
Anthropic	`Claude-User`	Fetches when a user asks Claude a question

Two practical notes from the docs:

OpenAI says sites opted out of OAI-SearchBot won't be shown in ChatGPT search answers (except as navigational links), and that a robots.txt change can take about 24 hours to take effect for search. Perplexity also mentions up to 24 hours.
User-triggered fetchers (ChatGPT-User, Perplexity-User) are documented as acting on behalf of a user, so robots.txt rules may not apply to them in the same way.

Sources: OpenAI crawlers, Perplexity crawlers, Anthropic crawlers.

2. Find AI bot hits in your access log

The examples assume the standard nginx/Apache "combined" log format, where the client IP is the first field, the status code is the ninth, and the user agent is the sixth field when you split on double quotes.

Count requests per bot:

grep -oE 'OAI-SearchBot|GPTBot|ChatGPT-User|PerplexityBot|Perplexity-User|Claude-SearchBot|ClaudeBot|Claude-User' \
  /var/log/nginx/access.log | sort | uniq -c | sort -rn

See which pages a specific bot requested:

grep 'OAI-SearchBot' /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

If your logs are rotated and gzipped, use zgrep on access.log.*.gz the same way.

3. Check the status codes — this is where problems hide

A bot that "visits" but receives 403 or 429 isn't reading your content. This is very common on sites behind a CDN or WAF with aggressive bot protection.

grep -E 'OAI-SearchBot|PerplexityBot|Claude-SearchBot' /var/log/nginx/access.log \
  | awk '{print $9}' | sort | uniq -c | sort -rn

What to look for:

200 on real content pages — good.
301/302 — fine if they resolve to a 200; check that redirects don't loop or land on a login page.
403 — usually a firewall or bot-protection rule. Perplexity's docs, for example, describe allowing their bots in Cloudflare or AWS WAF by combining a user-agent match with their published IP ranges.
429 — rate limiting. Bots may back off and come back less often.
5xx — your server failed; the bot got nothing useful.

Important: if a CDN blocks a request at the edge, it may never reach your origin server logs at all. In that case, check the CDN's own firewall/security events too.

4. Verify the IP, not just the user agent

OpenAI and Perplexity publish JSON files with the IP ranges their bots use:

https://openai.com/searchbot.json
https://openai.com/gptbot.json
https://openai.com/chatgpt-user.json
https://www.perplexity.ai/perplexitybot.json
https://www.perplexity.ai/perplexity-user.json

Each file contains a prefixes list with ipv4Prefix or ipv6Prefix entries. Here's a small script that checks IPs from your log against them:

#!/usr/bin/env python3
"""verify_ai_bots.py — check which logged AI-bot IPs match official ranges."""
import ipaddress, json, re, sys, urllib.request

SOURCES = {
    "OAI-SearchBot": "https://openai.com/searchbot.json",
    "GPTBot": "https://openai.com/gptbot.json",
    "ChatGPT-User": "https://openai.com/chatgpt-user.json",
    "PerplexityBot": "https://www.perplexity.ai/perplexitybot.json",
    "Perplexity-User": "https://www.perplexity.ai/perplexity-user.json",
}

def load_ranges(url):
    req = urllib.request.Request(url, headers={"User-Agent": "range-check"})
    data = json.load(urllib.request.urlopen(req, timeout=20))
    nets = []
    for p in data.get("prefixes", []):
        cidr = p.get("ipv4Prefix") or p.get("ipv6Prefix")
        if cidr:
            nets.append(ipaddress.ip_network(cidr))
    return nets

ranges = {bot: load_ranges(url) for bot, url in SOURCES.items()}
seen = {}

with open(sys.argv[1], errors="replace") as log:
    for line in log:
        for bot in ranges:
            if re.search(rf"\b{re.escape(bot)}\b", line):
                ip = line.split()[0]
                seen.setdefault((bot, ip), 0)
                seen[(bot, ip)] += 1

for (bot, ip), hits in sorted(seen.items()):
    addr = ipaddress.ip_address(ip)
    ok = any(addr in net for net in ranges[bot])
    print(f"{'OK  ' if ok else 'FAKE?'} {bot:16} {ip:40} {hits} hits")

Run it:

python3 verify_ai_bots.py /var/log/nginx/access.log

Anything marked FAKE? claims to be an AI bot but comes from an IP outside the published ranges. Don't panic: the ranges change over time, so re-download them before drawing conclusions. But if a "GPTBot" is hammering your site from a random hosting provider, it's very likely not OpenAI.

Note: if your server sits behind a reverse proxy or CDN, the first log field may be the proxy's IP. Make sure you log the real client IP (for example via X-Forwarded-For / CF-Connecting-IP with a trusted-proxy config).

5. For Googlebot and Bingbot: reverse DNS

Google and Bing also publish IP lists (googlebot.json, bingbot.json), and both support the classic reverse-then-forward DNS check:

host 66.249.66.1
# → ...googlebot.com
host crawl-66-249-66-1.googlebot.com
# → must resolve back to 66.249.66.1

If the reverse lookup doesn't end in a Google-owned domain (such as googlebot.com or google.com), or the forward lookup doesn't return the same IP, it isn't Googlebot. The same idea works for Bingbot with search.msn.com.

6. What a verified visit does — and doesn't — prove

A verified OAI-SearchBot hit with a 200 tells you one thing: the bot could fetch that page. It does not prove that:

the page is indexed in that assistant's search,
the page will be cited in an answer,
anyone will click through.

Those are separate stages, and each needs its own evidence: a saved answer that links to your page, and a real referral visit in your analytics. Treat bot logs as the first link in that chain, not the finish line.

Quick checklist

[ ] List AI bots in your logs and count hits per bot
[ ] Check status codes for search bots — no 403/429 on public pages
[ ] Check CDN/WAF events for edge blocks that never reach your origin
[ ] Verify IPs against the official JSON ranges (re-download them regularly)
[ ] Use reverse DNS for Googlebot and Bingbot
[ ] Don't treat a crawl as a citation

If you'd rather not script this by hand, AURA GEO runs these accessibility checks for your site, keeps a crawl history for AI bots and shows real referral visits from AI assistants — and your own agent can pick up the fixes over MCP. There's also an English write-up on why a bot visit isn't a citation.

How to Get Your Site Ready for AI Search: robots.txt, llms.txt, Schema and AI Bots — A Practical Checklist

AURA GEO — Tue, 20 Oct 2026 17:07:16 +0000

More and more people skip the ten blue links and search directly in ChatGPT, Gemini, Perplexity or Claude. An assistant with web search finds pages on its own, reads them and builds an answer with links to its sources. If your site is blocked, slow or hard to parse for these systems, it won't make it into the answer — even if it ranks well on Google.

Preparing a site for this kind of search is called GEO (Generative Engine Optimization). The good news: the technical foundation of GEO overlaps almost entirely with good SEO. The bad news: there are a few places where sites break all the time, and a standard SEO audit doesn't catch them.

Below is a checklist you can get through in an evening. Every item can be checked with a terminal command or a couple of lines of code.

1. Allow AI bots in robots.txt

The first thing any crawler does is read robots.txt. Over the past few years, plenty of sites copied "protective" rules and accidentally locked out AI search bots.

It's important to tell two kinds of bots apart:

Training bots (e.g. GPTBot, ClaudeBot, Google-Extended) collect data to train models.
Search and user-triggered bots (e.g. OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) show up when an assistant is looking for an answer for a specific user. These are the ones that bring citations and clicks.

You can block training and still allow search. Example:

# AI search and user-triggered bots — allow
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-User
Allow: /

# Classic search engines
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Check: make sure the file returns a 200 and doesn't contain a blanket Disallow: / for the agents you care about.

curl -s -o /dev/null -w "%{http_code}\n" https://example.com/robots.txt
curl -s https://example.com/robots.txt | grep -iE "user-agent|disallow"

Don't forget your CDN and WAF

A common trap: robots.txt allows everything, but Cloudflare or another WAF serves bots a 403 or a captcha. Check the server response with different User-Agents:

for ua in "OAI-SearchBot" "ChatGPT-User" "PerplexityBot" "Googlebot"; do
  code=$(curl -s -o /dev/null -w "%{http_code}" -A "$ua" https://example.com/)
  echo "$ua -> $code"
done

If anything isn't a 200, look at your bot-protection rules.

2. Serve text in HTML, not just via JavaScript

Many AI crawlers don't execute JavaScript, or only do so in a limited way. If your content only appears after an SPA hydrates, the bot sees an empty <div id="root">.

A simple check — download the page and see whether the main text is there:

curl -s https://example.com/pricing \
  | sed -e 's/<[^>]*>//g' | tr -s ' \n' | head -c 1500

If you see prices, descriptions and the FAQ, you're good. If all you get is the menu and empty space, you need SSR, SSG or prerendering (Next.js, Nuxt, Astro, Remix, etc.).

3. Add llms.txt

llms.txt is a proposed format: a Markdown file at the site root that briefly explains what the site is and lists key pages with descriptions. It isn't a standard, and no major assistant officially promises to use it. But it's cheap, useful for agents and developers, and it forces you to put the essentials of your product into words.

Minimal example:

# Example Inc.

> Example is an online booking service for small yoga studios.
> Operates in Latvia and Lithuania, interface in Russian and English.

## Main pages
- [Pricing](https://example.com/pricing): three plans, free trial.
- [API docs](https://example.com/docs/api): REST API for client bookings.
- [About](https://example.com/about): team, contacts, company details.

## FAQ
- [FAQ](https://example.com/faq): refunds, payments, integrations.

A good live example is AURA GEO's llms.txt: besides the list of pages, it honestly states what the service does not do (for instance, that it doesn't "index sites in AI directly"). Explicit limitations like this are useful — models have less to fill in on their own.

4. Schema.org markup (JSON-LD)

Structured data helps search engines — and, indirectly, AI — understand who you are and what you sell. The minimum set for a SaaS or service:

Organization — name, logo, sameAs with links to external profiles;
WebSite — site name;
SoftwareApplication or Product — for the product and pricing;
FAQPage — for a Q&A block;
Article — for blog posts and knowledge-base articles.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Example Inc.",
  "url": "https://example.com",
  "logo": "https://example.com/logo.png",
  "sameAs": [
    "https://github.com/example",
    "https://www.producthunt.com/products/example"
  ]
}
</script>

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [{
    "@type": "Question",
    "name": "Is there a free plan?",
    "acceptedAnswer": {
      "@type": "Answer",
      "text": "Yes, you can connect one site for free."
    }
  }]
}
</script>

The golden rule: markup must match the visible text. If your JSON-LD contains an FAQ that isn't on the page, that's worse than having no markup at all.

Check it with Google's Rich Results Test or validator.schema.org.

5. Sitemap, canonical and noindex

AI assistants with search lean heavily on the Google and Bing indexes. A page that isn't indexed has almost no chance of making it into an answer. Check:

# sitemap is reachable
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/sitemap.xml

# no accidental noindex, and canonical points to itself
curl -s https://example.com/pricing | grep -iE 'noindex|rel="canonical"'
curl -sI https://example.com/pricing | grep -i x-robots-tag

Also go into Google Search Console and Bing Webmaster Tools and review the report on non-indexed pages. Bing matters especially: several AI assistants rely on its index.

6. IndexNow: report updates right away

IndexNow is a protocol for notifying Bing, Yandex and other participating search engines about new or changed URLs. Google doesn't use it, but for Bing it's the fastest route.

# the key must be available at https://example.com/<KEY>.txt
curl -X POST "https://api.indexnow.org/indexnow" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "host": "example.com",
    "key": "<KEY>",
    "urlList": ["https://example.com/pricing", "https://example.com/faq"]
  }'

Keep in mind: a successful submission only means "notification received" — not "page indexed", and certainly not "AI is citing it". These are separate stages, and each is worth checking on its own.

7. Write pages that are easy to cite

Technical work opens the door, but content is what gets cited. What works:

Answer in the first paragraph. Question in the heading, a direct 2–3 sentence answer right below it.
Specifics. Prices, timelines, countries, limitations, versions. "Free for one site" beats "affordable pricing".
Verifiable facts. Last-updated date, author, source of the numbers.
Structure. H2/H3, lists, comparison tables.
One page per question. A "How much does X cost" page gets cited more than a paragraph buried in a giant landing page.
Don't pass off demo data as facts. If your landing page has nice example numbers in an interface mockup, label them as examples — otherwise a model may repeat them as real.

8. Measure: bot logs and AI referrals

Without measurement, GEO turns into guesswork. The minimum:

Server logs. Look for AI bot visits:

grep -iE "GPTBot|OAI-SearchBot|ChatGPT-User|PerplexityBot|ClaudeBot|Claude-User" \
  /var/log/nginx/access.log | awk '{print $1, $7, $12}' | tail -50

Referrers in analytics. Visits come from domains like chatgpt.com, perplexity.ai, gemini.google.com, copilot.microsoft.com. Create a separate segment for them.
Manual answer checks. Ask your customers' neutral questions in fresh chats with search enabled, save the answer and the list of sources, and repeat after making changes. A single check proves nothing — the trend is what matters.

You can stitch all of this together with scripts, or keep it in one dashboard. For example, AURA GEO runs a technical audit (robots.txt, accessible HTML, noindex, sitemap), counts recognized AI referrals, lets you hand tasks to your own AI agent via MCP, and submits updated pages through IndexNow. The basic plan for one site is free.

Final checklist

Item	How to check
robots.txt allows AI search bots	`curl robots.txt`, grep for User-agent
WAF doesn't block bots	`curl -A "ChatGPT-User"` → 200
Text is in HTML without JS	`curl` + strip tags
llms.txt at the root	`curl /llms.txt`
JSON-LD: Organization, FAQPage	Rich Results Test
Sitemap, canonical, no noindex	grep + Search Console / Bing WMT
IndexNow configured	200/202 response from the API
Pages answer specific questions	content review
Bot logs and AI referrers are tracked	log grep, analytics segment

FAQ

What is GEO in plain terms?

GEO (Generative Engine Optimization) is preparing a site so that AI assistants with web search can find, read and cite its pages. SEO is about rankings in regular search results; GEO is about showing up in answers from ChatGPT, Gemini, Perplexity and others.

Should I block GPTBot to protect my content?

That's your call. Blocking training bots (GPTBot, ClaudeBot, Google-Extended) doesn't affect search bots like OAI-SearchBot and ChatGPT-User. If you want citations, don't block search and user-triggered bots.

Is llms.txt required?

No. It's an optional format, and major assistants don't officially promise to use it. But it's cheap, useful for AI agents and developers, and helps you describe your product clearly.

Does schema.org help you get into AI answers?

There are no direct guarantees, but markup helps search engines understand the entities on a page, and AI search relies heavily on their indexes. The key is that markup matches the visible text.

Does submitting to IndexNow mean the page will show up in ChatGPT?

No. IndexNow only notifies search engines. Indexing, a bot visit, a citation and a click are separate stages, and each needs to be checked separately.

How do I know AI is already sending traffic?

Look for chatgpt.com, perplexity.ai and gemini.google.com referrers in your analytics and AI bot visits in your logs. And regularly check how assistants answer your customers' questions.