Check your website’s access logs for “bot,” and you’ll likely find more than just Googlebot. GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot may also be visiting your pages, sometimes thousands of times a week.
It’s easy to group them all under “AI crawlers,” but they don’t all work the same way. Some collect content for AI training, others help AI search engines find and index pages, and some visit a specific URL when someone shares it in a chat.
That difference matters when deciding which bots to allow or block. You could accidentally keep your site out of ChatGPT search results while still allowing your content to be used for AI training.
In this post, we’ll cover:
- The three main types of AI crawlers and what each one does
- The user agents used by OpenAI, Anthropic, Perplexity, and Google
- Why many AI crawlers don’t execute JavaScript and what that means for your content
- How to identify AI crawler activity in your server logs using a few simple commands
- How to configure robots.txt to restrict AI training crawlers while keeping your content accessible to AI search
The Three Main Types of AI Crawlers
Not every AI bot visits your website for the same reason. Some collect content for training, others help AI assistants find information, and a few only visit when someone asks about a specific page.
Understanding these differences helps you decide which bots to allow and which ones to block.
1. Training Crawlers
Training crawlers collect website content that may be used to train future AI models. Two common examples are GPTBot from OpenAI and ClaudeBot from Anthropic.
Both companies allow website owners to opt out of training-related crawling through robots.txt.
But there are two things worth knowing:
- Blocking these bots doesn't remove content they've already collected.
- Blocking training crawlers doesn't automatically prevent your website from appearing in AI search results.
- So, if you don't want your content used for AI training, you can restrict these crawlers without necessarily losing visibility in AI search.
2. Search Crawlers
Search crawlers help AI assistants find relevant pages when someone asks a question. They work in a similar way to traditional search engine crawlers like Googlebot.
Some examples include:
- OAI-SearchBot: Used by ChatGPT search
- Claude-SearchBot: Used by Claude's search features
- PerplexityBot: Used by Perplexity
If you block these bots, you may limit your website's chances of appearing as an AI citation in their search-generated answers.
OpenAI also notes that changes to robots.txt can take around 24 hours to be reflected in ChatGPT search.
If AI search visibility matters to your business, these are the crawlers you'll generally want to allow.
3. User-Triggered Fetchers
These bots work a little differently. Rather than regularly crawling websites, they visit a page when someone specifically asks an AI assistant to access it.
For example, someone might paste your website URL into ChatGPT and ask, "Can you summarize this page?" or "What does this product cost?"
The assistant may then use a fetcher to retrieve that page.
Common examples include ChatGPT-User, Claude-User, and Perplexity-User.
There's an important difference in how these bots handle robots.txt. OpenAI and Perplexity indicate that robots.txt restrictions may not apply to certain user-initiated requests. Anthropic, on the other hand, says its bots follow robots.txt rules.
What Does This Mean for Your Website?
You don't have to allow every AI crawler or block them all.
OpenAI and Anthropic use separate crawlers for training and search, so you can manage access to each independently.
For example, if you want your pages to appear in AI search results but don't want them collected for model training, you can block training crawlers while allowing search crawlers.
The key is knowing what each bot does before adding it to your robots.txt file.
Who's Who: AI Crawler User Agents You Should Know
Each AI crawler has a user-agent token that identifies it. This is what you use when setting rules in your robots.txt file.
The full user-agent string may change as bots are updated. For example, GPTBot/1.3 might become GPTBot/1.4, but the token GPTBot stays the same.
Here's a breakdown of the main crawlers, what they do, and how to verify them.
| Company | Token | Job | Follows robots.txt? | Verify by IP |
|---|---|---|---|---|
| OpenAI | GPTBot |
Training | Yes | openai.com/gptbot.json |
| OpenAI | OAI-SearchBot |
ChatGPT search index | Yes | openai.com/searchbot.json |
| OpenAI | ChatGPT-User |
User-triggered fetch | "May not apply" | openai.com/chatgpt-user.json |
| Anthropic | ClaudeBot |
Training | Yes | No published ranges |
| Anthropic | Claude-SearchBot |
Claude search index | Yes | No published ranges |
| Anthropic | Claude-User |
User-triggered fetch | Yes | No published ranges |
| Perplexity | PerplexityBot |
Search index (not training) | Yes | perplexity.com/perplexitybot.json |
| Perplexity | Perplexity-User |
User-triggered fetch | Generally no | perplexity.com/perplexity-user.json |
Googlebot |
Search, including AI Overviews | Yes | Google's published ranges | |
Google-Extended |
Gemini training and grounding (a token, not a bot) | n/a | n/a |
A Few Things Worth Knowing
Google-Extended won't appear in your server logs. It's a robots.txt control token, not a separate crawler. Google uses its existing crawlers to access pages and checks Google-Extended rules to determine whether content can be used for certain Gemini-related purposes.
Blocking Google-Extended doesn't affect your regular Google Search rankings. It also doesn't block your pages from appearing in AI Overviews, which rely on Google's search crawling infrastructure.
Don't trust a user-agent name without checking it. Anyone can configure a bot to identify itself as GPTBot or Googlebot. If you notice unusually heavy traffic from one of these crawlers, check its IP address against the operator's published ranges before deciding what to do.
There are more AI-related crawlers than the ones listed above. You may also come across Meta-ExternalAgent, ByteDance's Bytespider, Amazonbot, or Applebot. Each has its own purpose and crawling policies, so check the official documentation before allowing or blocking them.
OpenAI also operates OAI-AdsBot, which visits landing pages associated with ChatGPT ads. Unless you're running those ads, you're unlikely to encounter it.
Most AI Crawlers Don't Execute JavaScript
This is especially important if your website relies on JavaScript to display its content.
A study by Vercel and MERJ examined AI crawler activity across Vercel's network and found no evidence that several major AI crawlers executed JavaScript.
That included crawlers from OpenAI, Anthropic, Meta, ByteDance, and Perplexity.
Interestingly, GPTBot and ClaudeBot did request JavaScript files. Around 11.5% of GPTBot's requests and 23.8% of ClaudeBot's requests were for JavaScript files, but the study found no evidence that they actually executed that code.
Google's Gemini-related search capabilities are different because Googlebot has access to Google's rendering infrastructure. Applebot also uses browser-based rendering.
Why This Matters for Your Website
Think about a pricing page built entirely with client-side JavaScript.
When someone opens it in a browser, everything looks fine. They can see the pricing plans, product details, and other content once JavaScript finishes loading.
But a crawler that doesn't execute JavaScript may receive only something like this:
<body>
<div id="root"></div>
<script src="/app.js"></script>
</body>
The actual page content isn't present in the initial HTML response.
That means the crawler may not be able to access the information you want it to read, even though the page works perfectly for visitors.
This can create a serious AI crawlability issue, especially for important pages such as product descriptions, pricing, documentation, and service pages.
How to Check Whether AI Crawlers Can Read Your Content
You can run a simple command to see what your server returns without executing JavaScript.
curl -s -A "Mozilla/5.0 (compatible; GPTBot/1.3; +https://openai.com/gptbot)" \
https://yoursite.com/pricing \
| sed -n '/<body/,/<\/body>/p' | head -40
Replace https://yoursite.com/pricing with the URL you want to check.
Look at the HTML returned by the command.
- If you can see your main heading, pricing details, and page copy: Your important content is available in the initial HTML response.
- If you only see an empty root element and script tags: Your content may depend on JavaScript, making it harder for non-rendering crawlers to access.
Keep in mind that this test checks the HTML your server returns. It doesn't confirm whether a particular bot will index or use the page.
If your important content is missing, consider server-side rendering (SSR), static site generation (SSG), or prerendering.
Frameworks like Next.js, Nuxt, Astro, and SvelteKit support these approaches. Depending on how your website is built, you may be able to fix the issue without rebuilding the entire site.
Don't Overlook Broken URLs and Redirects
The Vercel and MERJ study also found that AI crawlers spent a noticeable amount of time requesting pages that no longer existed or following redirects.
According to the study, around 34% of ChatGPT and Claude crawler fetches returned 404 errors. ChatGPT also spent roughly 14% of its requests following redirects.
That's a good reason to check the URLs you're making available to crawlers.
Make sure your XML sitemap contains working, canonical URLs. Fix broken internal links, remove outdated URLs from your sitemap, and avoid unnecessary redirect chains.
These are basic technical SEO practices, but they also make it easier for AI crawlers to reach the content that matters.
One thing to keep in mind: The Vercel study used data from December 2024. AI crawlers continue to change, so don't assume every bot behaves exactly the same way today.
The safest approach is to test your own pages, review your server logs, and make sure your important content is accessible without relying entirely on JavaScript.
Find them in your own logs
Your access logs already hold the answer. These commands assume the standard nginx/Apache "combined" log format; adjust the field numbers if yours differs.
Which AI bots visit, and how often:
bash
grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" \
/var/log/nginx/access.log | sort | uniq -c | sort -rn
Which pages a specific bot requests most:
bash
grep "OAI-SearchBot" /var/log/nginx/access.log \
| awk '{print $7}' | sort | uniq -c | sort -rn | head -20
What status codes it gets back (lots of 404s or 301s means wasted crawls):
bash
grep "GPTBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c
Whether it's getting an empty shell: sort by response size. A 200 with a 2 KB body on a page that's 40 KB in the browser is the JavaScript problem from the previous section.
bash
grep "ClaudeBot" /var/log/nginx/access.log | awk '{print $10, $7}' | sort -n | head -20
If you're behind Cloudflare or another CDN, its bot analytics may show this without touching logs, and it usually separates verified bots from spoofed ones.
Keep in mind what this tells you: who came and what they fetched. It doesn't tell you whether any of it ended up cited in an answer. That takes a different kind of measurement, which we cover in how to track AI search visibility.
A robots.txt that opts out of training but stays in AI search
A common goal is no training on your content, while staying citable in ChatGPT, Claude and Perplexity search. Here's a setup for that:
text
# Opt out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
# Stay in AI search indexes
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /admin/
Allow: /
# Everyone else
User-agent: *
Disallow: /admin/`
Sitemap: https://yoursite.com/sitemap.xml
Three gotchas that catch people:
- A bot only reads the most specific group that matches it. Once OAI-SearchBot has its own group, it ignores everything under User-agent: *. That's why Disallow: /admin/ is repeated above. Forget it and the search bots can crawl your admin paths.
- robots.txt is per host. blog.yoursite.com and docs.yoursite.com each need their own file. Anthropic calls this out explicitly in its crawler docs.
- Google-Extended has a trade-off. Blocking it also opts you out of Gemini grounding, the live sources Gemini cites. If Gemini answers matter to you, you may want to allow it and accept the training use. It doesn't affect Google Search or AI Overviews either way.
If you'd rather slow a bot down than block it, Anthropic supports the non-standard Crawl-delay directive (Crawl-delay: 1 under User-agent: ClaudeBot).
More background on these rules is in our glossary entry on robots.txt for AI crawlers, and we'll go deeper on it in the next post in this series.
Wrap-up
- "AI crawler" covers three different jobs: training, search indexing, and user-triggered fetches. Treat them separately.
- At OpenAI and Anthropic, training and search are separate switches. You can opt out of one and keep the other.
- Most AI crawlers don't run JavaScript. If your content only exists after hydration, they can't see it. curl your own pages to check.
- Your access logs already show which bots visit, what they fetch and what status codes they get.
- Verify by IP before trusting a user-agent string.
Next in the series: robots.txt for AI crawlers in detail, then what llms.txt is and whether you need one.
Which of these bots shows up most in your logs? I'm curious whether anyone's seeing Claude-SearchBot or PerplexityBot outpace GPTBot yet.
Originally published on the Track My Visibility blog. This post was drafted with AI assistance and reviewed and tested by our team.
Top comments (0)