Most site owners have a robots.txt that mentions Googlebot and maybe Bingbot, and nothing else. But there are now at least 15 AI crawlers hitting your site — some collecting training data, some powering live answers in ChatGPT and Perplexity — and they do not all behave the way their documentation claims.
I spent a while digging into how each one actually behaves and building free tools to check it, and a few things genuinely surprised me. Here's the rundown.
The two jobs AI crawlers do (and why blocking one doesn't block the other)
Before the compliance stuff, the single most common misconception: "blocking ChatGPT."
AI companies run separate crawlers for separate jobs, under different user-agent strings:
-
Training crawlers collect content to train future models.
GPTBot(OpenAI),ClaudeBot(Anthropic),CCBot(Common Crawl). -
Search / live crawlers fetch pages so an assistant can cite them in a live answer.
OAI-SearchBot,PerplexityBot,ChatGPT-User.
So when someone adds this to block "ChatGPT":
User-agent: GPTBot
Disallow: /
...they've opted out of training, but they're still fully visible in ChatGPT's search answers, which use OAI-SearchBot. Different pipeline, different user agent. Blocking one has zero effect on the other.
The compliance findings
Here's where it gets interesting. Not every crawler honors robots.txt, and the docs don't always match reality:
| Crawler | Operator | Respects robots.txt? |
|---|---|---|
| GPTBot | OpenAI | ✅ Yes (documented) |
| OAI-SearchBot | OpenAI | ✅ Yes |
| ClaudeBot | Anthropic | ✅ Yes |
| CCBot | Common Crawl | ✅ Yes |
| Google-Extended | Google | ✅ Yes (it's a token, not a crawler — more below) |
| Applebot-Extended | Apple | ✅ Yes (also a token) |
| PerplexityBot | Perplexity | ⚠️ Partial — Cloudflare reported undeclared fetching that bypassed blocks |
| Perplexity-User | Perplexity | ❌ No — skips robots.txt by design |
| Bytespider | ByteDance | ❌ No — widely reported to ignore it entirely |
| cohere-ai | Cohere | ❓ Undocumented — can't verify |
Three things worth calling out:
1. Bytespider (ByteDance) just doesn't care. No published documentation, crawls aggressively, and multiple independent reports show it fetching pages on sites that disallowed it. If you actually want it gone, robots.txt won't do it — you need a firewall/CDN rule matching the user agent.
2. Perplexity-User ignores robots.txt on purpose. Perplexity's own docs say so: because the fetch is triggered directly by a user asking a question, they treat it like a browser, not a crawler. Reasonable framing, but it means "block Perplexity via robots.txt" is only half true.
3. Google-Extended and Applebot-Extended aren't crawlers at all. They're product tokens. You'll never see them in your logs. Regular Googlebot/Applebot fetches your pages as always; the token just governs whether that already-crawled content may train Gemini / Apple Intelligence. Which leads to the mistake I see most often...
The mistake: thinking Google-Extended controls AI Overviews
It doesn't. Blocking Google-Extended:
- ✅ Opts you out of Gemini training
- ❌ Does nothing to AI Overviews (those are a Search feature, built from the Search index)
- ❌ Has zero effect on your Google rankings
There's no clean "opt out of AI Overviews" switch — the only levers are
nosnippet/max-snippet, which also shrink your normal search snippets. So blocking Google-Extended is a rare free opt-out: you lose nothing in Search, you just leave Gemini's training set. ## The other thing that quietly breaks AI visibility: JavaScript Here's one that catches SPAs hard. Most AI crawlers don't execute JavaScript. Unlike Googlebot — which has a full headless-Chrome rendering pipeline — GPTBot, ClaudeBot, and PerplexityBot mostly fetch your raw HTML and move on. So if your content is client-side rendered:
html
<body>
<div id="root"></div>
<!-- content injected by JS after load -->
</body>
...that's roughly what the AI web sees of your site: an empty shell. The quick test:
curl -s https://yoursite.com/your-page | grep -i "a phrase from your main content"
If the phrase isn't in the response, it isn't in your server-rendered HTML, and AI crawlers can't read it. The fix is SSR/SSG (Next.js, Nuxt, etc.) or pre-rendering for bot user agents. Google is basically the only exception here — assume everything else reads raw HTML only.
What I built out of all this
I got tired of not being able to answer basic questions — can ChatGPT even see this site? which bots am I accidentally blocking? — and the existing "AI SEO" tools are mostly $30–500/mo SaaS. So I built free browser-based versions of the pieces I needed and put them at geoprompttracker.com:
A directory of every AI crawler with the compliance notes above and copy-paste allow/block rules per bot
An AI Crawler Access Checker that reads your live robots.txt and shows which of the 15 bots you allow or block
An llms.txt generator/validator for the emerging llms.txt convention
A robots.txt generator with per-bot toggles (block training, allow search — or whatever split you want)
Stack, for the curious: Next.js App Router on Vercel, everything static or client-side, no accounts, no signup. The few tools that fetch a URL go through a rate-limited proxy with an SSRF guard; nothing is stored.
The 5-minute audit, if you do nothing else
Decide your policy: training vs. search, per operator. They're separate choices.
Check what your robots.txt currently says with a crawler checker — a lot of "block all AI" configs accidentally kill Perplexity/OAI-SearchBot, which are the ones that send you traffic.
curl a key page and confirm your content is in the raw HTML, not JS-injected.
For the bots that ignore robots.txt (Bytespider, Perplexity-User), use firewall rules if you actually care.
I'd genuinely like feedback on the bot-compliance data — if you've got server logs showing different behavior (especially anything that's stopped respecting robots.txt recently), I'd love to hear it. This space changes month to month and I'm refreshing the list as it does.
---
### Two small notes before you hit publish
1. **Leave `canonical_url` unset.** This article is original to dev.to, so let dev.to be canonical — you still get the backlinks in the body + referral traffic + dev.to's own Google ranking pointing readers to you. (Only set a canonical if you were reposting something already on your site.)
2. **Tags:** I used `webdev, ai, seo, programming` — all have real audiences on dev.to. You can swap `programming` for `nextjs` if you'd rather reach that crowd.
After it's live, drop the link in the r/TechSEO post too — the article and the `/bots` directory reinforce each other. Want me to draft the **r/TechSEO post** next so you can fire both today while the HN account warms up?
Top comments (0)