DEV Community: Track My Visibility

How AI Crawlers Actually Work: GPTBot, ClaudeBot and PerplexityBot

Track My Visibility — Fri, 30 Oct 2026 09:27:45 +0000

Check your website’s access logs for “bot,” and you’ll likely find more than just Googlebot. GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot may also be visiting your pages, sometimes thousands of times a week.

It’s easy to group them all under “AI crawlers,” but they don’t all work the same way. Some collect content for AI training, others help AI search engines find and index pages, and some visit a specific URL when someone shares it in a chat.

That difference matters when deciding which bots to allow or block. You could accidentally keep your site out of ChatGPT search results while still allowing your content to be used for AI training.

In this post, we’ll cover:

The three main types of AI crawlers and what each one does
The user agents used by OpenAI, Anthropic, Perplexity, and Google
Why many AI crawlers don’t execute JavaScript and what that means for your content
How to identify AI crawler activity in your server logs using a few simple commands
How to configure robots.txt to restrict AI training crawlers while keeping your content accessible to AI search

The Three Main Types of AI Crawlers

Not every AI bot visits your website for the same reason. Some collect content for training, others help AI assistants find information, and a few only visit when someone asks about a specific page.
Understanding these differences helps you decide which bots to allow and which ones to block.

1. Training Crawlers

Training crawlers collect website content that may be used to train future AI models. Two common examples are GPTBot from OpenAI and ClaudeBot from Anthropic.

Both companies allow website owners to opt out of training-related crawling through robots.txt.

But there are two things worth knowing:

Blocking these bots doesn't remove content they've already collected.
Blocking training crawlers doesn't automatically prevent your website from appearing in AI search results.
So, if you don't want your content used for AI training, you can restrict these crawlers without necessarily losing visibility in AI search.

2. Search Crawlers

Search crawlers help AI assistants find relevant pages when someone asks a question. They work in a similar way to traditional search engine crawlers like Googlebot.

Some examples include:

OAI-SearchBot: Used by ChatGPT search
Claude-SearchBot: Used by Claude's search features
PerplexityBot: Used by Perplexity

If you block these bots, you may limit your website's chances of appearing as an AI citation in their search-generated answers.

OpenAI also notes that changes to robots.txt can take around 24 hours to be reflected in ChatGPT search.

If AI search visibility matters to your business, these are the crawlers you'll generally want to allow.

3. User-Triggered Fetchers

These bots work a little differently. Rather than regularly crawling websites, they visit a page when someone specifically asks an AI assistant to access it.

For example, someone might paste your website URL into ChatGPT and ask, "Can you summarize this page?" or "What does this product cost?"

The assistant may then use a fetcher to retrieve that page.

Common examples include ChatGPT-User, Claude-User, and Perplexity-User.

There's an important difference in how these bots handle robots.txt. OpenAI and Perplexity indicate that robots.txt restrictions may not apply to certain user-initiated requests. Anthropic, on the other hand, says its bots follow robots.txt rules.

What Does This Mean for Your Website?

You don't have to allow every AI crawler or block them all.

OpenAI and Anthropic use separate crawlers for training and search, so you can manage access to each independently.

For example, if you want your pages to appear in AI search results but don't want them collected for model training, you can block training crawlers while allowing search crawlers.

The key is knowing what each bot does before adding it to your robots.txt file.

Who's Who: AI Crawler User Agents You Should Know

Each AI crawler has a user-agent token that identifies it. This is what you use when setting rules in your robots.txt file.

The full user-agent string may change as bots are updated. For example, GPTBot/1.3 might become GPTBot/1.4, but the token GPTBot stays the same.

Here's a breakdown of the main crawlers, what they do, and how to verify them.

Company	Token	Job	Follows robots.txt?	Verify by IP
OpenAI	`GPTBot`	Training	Yes	`openai.com/gptbot.json`
OpenAI	`OAI-SearchBot`	ChatGPT search index	Yes	`openai.com/searchbot.json`
OpenAI	`ChatGPT-User`	User-triggered fetch	"May not apply"	`openai.com/chatgpt-user.json`
Anthropic	`ClaudeBot`	Training	Yes	No published ranges
Anthropic	`Claude-SearchBot`	Claude search index	Yes	No published ranges
Anthropic	`Claude-User`	User-triggered fetch	Yes	No published ranges
Perplexity	`PerplexityBot`	Search index (not training)	Yes	`perplexity.com/perplexitybot.json`
Perplexity	`Perplexity-User`	User-triggered fetch	Generally no	`perplexity.com/perplexity-user.json`
Google	`Googlebot`	Search, including AI Overviews	Yes	Google's published ranges
Google	`Google-Extended`	Gemini training and grounding (a token, not a bot)	n/a	n/a

A Few Things Worth Knowing

Google-Extended won't appear in your server logs. It's a robots.txt control token, not a separate crawler. Google uses its existing crawlers to access pages and checks Google-Extended rules to determine whether content can be used for certain Gemini-related purposes.

Blocking Google-Extended doesn't affect your regular Google Search rankings. It also doesn't block your pages from appearing in AI Overviews, which rely on Google's search crawling infrastructure.

Don't trust a user-agent name without checking it. Anyone can configure a bot to identify itself as GPTBot or Googlebot. If you notice unusually heavy traffic from one of these crawlers, check its IP address against the operator's published ranges before deciding what to do.

There are more AI-related crawlers than the ones listed above. You may also come across Meta-ExternalAgent, ByteDance's Bytespider, Amazonbot, or Applebot. Each has its own purpose and crawling policies, so check the official documentation before allowing or blocking them.

OpenAI also operates OAI-AdsBot, which visits landing pages associated with ChatGPT ads. Unless you're running those ads, you're unlikely to encounter it.

Most AI Crawlers Don't Execute JavaScript

This is especially important if your website relies on JavaScript to display its content.

A study by Vercel and MERJ examined AI crawler activity across Vercel's network and found no evidence that several major AI crawlers executed JavaScript.

That included crawlers from OpenAI, Anthropic, Meta, ByteDance, and Perplexity.

Interestingly, GPTBot and ClaudeBot did request JavaScript files. Around 11.5% of GPTBot's requests and 23.8% of ClaudeBot's requests were for JavaScript files, but the study found no evidence that they actually executed that code.

Google's Gemini-related search capabilities are different because Googlebot has access to Google's rendering infrastructure. Applebot also uses browser-based rendering.

Why This Matters for Your Website

Think about a pricing page built entirely with client-side JavaScript.
When someone opens it in a browser, everything looks fine. They can see the pricing plans, product details, and other content once JavaScript finishes loading.

But a crawler that doesn't execute JavaScript may receive only something like this:

<body>
  <div id="root"></div>
  <script src="/app.js"></script>
</body>

The actual page content isn't present in the initial HTML response.
That means the crawler may not be able to access the information you want it to read, even though the page works perfectly for visitors.

This can create a serious AI crawlability issue, especially for important pages such as product descriptions, pricing, documentation, and service pages.

How to Check Whether AI Crawlers Can Read Your Content

You can run a simple command to see what your server returns without executing JavaScript.

curl -s -A "Mozilla/5.0 (compatible; GPTBot/1.3; +https://openai.com/gptbot)" \

  https://yoursite.com/pricing \
  | sed -n '/<body/,/<\/body>/p' | head -40

Replace https://yoursite.com/pricing with the URL you want to check.

Look at the HTML returned by the command.

If you can see your main heading, pricing details, and page copy: Your important content is available in the initial HTML response.
If you only see an empty root element and script tags: Your content may depend on JavaScript, making it harder for non-rendering crawlers to access.

Keep in mind that this test checks the HTML your server returns. It doesn't confirm whether a particular bot will index or use the page.
If your important content is missing, consider server-side rendering (SSR), static site generation (SSG), or prerendering.

Frameworks like Next.js, Nuxt, Astro, and SvelteKit support these approaches. Depending on how your website is built, you may be able to fix the issue without rebuilding the entire site.

Don't Overlook Broken URLs and Redirects

The Vercel and MERJ study also found that AI crawlers spent a noticeable amount of time requesting pages that no longer existed or following redirects.

According to the study, around 34% of ChatGPT and Claude crawler fetches returned 404 errors. ChatGPT also spent roughly 14% of its requests following redirects.

That's a good reason to check the URLs you're making available to crawlers.

Make sure your XML sitemap contains working, canonical URLs. Fix broken internal links, remove outdated URLs from your sitemap, and avoid unnecessary redirect chains.

These are basic technical SEO practices, but they also make it easier for AI crawlers to reach the content that matters.

One thing to keep in mind: The Vercel study used data from December 2024. AI crawlers continue to change, so don't assume every bot behaves exactly the same way today.

The safest approach is to test your own pages, review your server logs, and make sure your important content is accessible without relying entirely on JavaScript.

Find them in your own logs

Your access logs already hold the answer. These commands assume the standard nginx/Apache "combined" log format; adjust the field numbers if yours differs.

Which AI bots visit, and how often:

bash
grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" \
  /var/log/nginx/access.log | sort | uniq -c | sort -rn

Which pages a specific bot requests most:

bash
grep "OAI-SearchBot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

What status codes it gets back (lots of 404s or 301s means wasted crawls):

bash
grep "GPTBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c

Whether it's getting an empty shell: sort by response size. A 200 with a 2 KB body on a page that's 40 KB in the browser is the JavaScript problem from the previous section.

bash
grep "ClaudeBot" /var/log/nginx/access.log | awk '{print $10, $7}' | sort -n | head -20

If you're behind Cloudflare or another CDN, its bot analytics may show this without touching logs, and it usually separates verified bots from spoofed ones.

Keep in mind what this tells you: who came and what they fetched. It doesn't tell you whether any of it ended up cited in an answer. That takes a different kind of measurement, which we cover in how to track AI search visibility.

A robots.txt that opts out of training but stays in AI search

A common goal is no training on your content, while staying citable in ChatGPT, Claude and Perplexity search. Here's a setup for that:
text

# Opt out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Stay in AI search indexes
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /admin/
Allow: /

# Everyone else
User-agent: *
Disallow: /admin/`

Sitemap: https://yoursite.com/sitemap.xml

Three gotchas that catch people:

A bot only reads the most specific group that matches it. Once OAI-SearchBot has its own group, it ignores everything under User-agent: *. That's why Disallow: /admin/ is repeated above. Forget it and the search bots can crawl your admin paths.
robots.txt is per host. blog.yoursite.com and docs.yoursite.com each need their own file. Anthropic calls this out explicitly in its crawler docs.
Google-Extended has a trade-off. Blocking it also opts you out of Gemini grounding, the live sources Gemini cites. If Gemini answers matter to you, you may want to allow it and accept the training use. It doesn't affect Google Search or AI Overviews either way.

If you'd rather slow a bot down than block it, Anthropic supports the non-standard Crawl-delay directive (Crawl-delay: 1 under User-agent: ClaudeBot).

More background on these rules is in our glossary entry on robots.txt for AI crawlers, and we'll go deeper on it in the next post in this series.

Wrap-up

"AI crawler" covers three different jobs: training, search indexing, and user-triggered fetches. Treat them separately.
At OpenAI and Anthropic, training and search are separate switches. You can opt out of one and keep the other.
Most AI crawlers don't run JavaScript. If your content only exists after hydration, they can't see it. curl your own pages to check.
Your access logs already show which bots visit, what they fetch and what status codes they get.
Verify by IP before trusting a user-agent string.

Next in the series: robots.txt for AI crawlers in detail, then what llms.txt is and whether you need one.

Which of these bots shows up most in your logs? I'm curious whether anyone's seeing Claude-SearchBot or PerplexityBot outpace GPTBot yet.

Originally published on the Track My Visibility blog. This post was drafted with AI assistance and reviewed and tested by our team.

Top 10 On-Page SEO Practices You Should Implement in 2026

Track My Visibility — Mon, 16 Feb 2026 08:38:55 +0000

SEO isn’t what it used to be. A few years ago, ranking on Google was enough. Today, people increasingly ask AI tools like ChatGPT, Perplexity, and Gemini for answers instead of scrolling through search results. These tools don’t show ten links - they give a quick summary with only a few brand mentions.
That changes the game. It’s no longer just about keywords. Your pages need to be easy to understand, easy to summarize, and easy to trust. If AI can’t figure out what your page offers, it’s unlikely to reference it - even if you rank.
For marketing teams and SEO pros, this means rethinking what “good on-page SEO” looks like. The fundamentals still matter, but they’re no longer enough.

Here are ten on-page SEO practices to focus on in 2026.

1. Write for Search Intent - Not Just Keywords

Sprinkling keywords in titles and headings isn’t enough anymore. In 2026, search engines and AI care about whether your page actually solves the searcher’s problem.
Ask yourself before publishing: Are people looking to learn, compare, or take action? Make that clear right away.
For example, instead of “Marketing Automation Tools”, use “Best Marketing Automation Tools for B2B SaaS Companies in 2026”; it immediately shows who it’s for and what it covers.

Quick checks:

Is the intent clear?
Does the headline show who it’s for?
Is the main answer easy to find at the top? If readers have to guess, you’ve already lost them. Clear intent is the foundation of modern SEO.

2. Place the Primary Answer at the Top

One of the biggest mistakes pages still make is taking too long to get to the point. When someone searches for something specific, they want the answer immediately. Yet many pages start with long introductions or background context. That might feel thorough, but it’s frustrating for readers and for AI tools scanning your content.

If your main answer is buried halfway down the page, it’s less likely to be read, summarized, or cited.

The fix is simple: lead with the answer, then expand. Start with a clear, direct response to the main question, keeping it concise and easy to understand. Then use subheadings to provide examples, context, or supporting details. Skip the long warm-up.

A good test: if someone reads only the first 150 words, will they walk away with the main takeaway?

When your primary answer is upfront, your content becomes easier to trust, easier to scan, and more likely to show up in AI-driven results.

3. Optimize Title Tags for Precision and Context

Generic titles like “SEO Tips” don’t tell readers or AI who the content is for or what it covers.

In 2026, your title should quickly show:

Who it’s for
What it covers
When or in what context does it applies

For example, “10 On-Page SEO Practices for SaaS Companies in 2026” is clear, specific, and easy to understand at a glance.

Before publishing, ask:

Would someone know instantly that this is for them?
Is the context obvious without reading the article?
Does it stand out from similar pages?

Specific titles don’t just improve clicks - they help AI understand your page and boost visibility.

Clean Up Your Heading Structure (It Matters More Than You Think)

Messy headings can quietly hurt your SEO. Multiple H1s, skipped levels, or using headings just to make text bigger might look fine to readers, but confuse search engines and AI.

Headings aren’t just formatting - they show structure. At a minimum:

Use one clear H1 for the main topic
Use H2s for major sections
Use H3s to break down those sections
Don’t choose heading levels based on size alone

A clear hierarchy makes your content easier to scan and easier for AI systems to understand - which means your page is more likely to be referenced correctly.

5. Use Structured Data (Schema Markup)

Structured data is often overlooked, but it helps search engines and AI understand what your page is about.

You can use it for things like:

Products
Services
Reviews
FAQs
Organizations
Authors

For example:

Add Product schema to pricing pages
Add FAQ schema to help sections
Add the Organization schema to your homepage

It doesn’t change how your page looks, but it makes your content easier for machines to read and increases your chances of rich results and AI visibility.

6. Define Your Brand as a Clear Entity

AI search tools rely on understanding your brand as an “entity,” so your website needs to clearly say who you are and what you do. Avoid vague statements like “We provide innovative solutions.”

Be specific:

Who you are
What you do
Who you serve
Where you operate
What makes you different

For example: “We provide compliance automation software for mid-sized fintech companies.”

Inconsistent or unclear messaging across pages can dilute entity recognition, even if individual pages are well optimized. This is where validation matters. Tools like Track My Visibility help identify whether AI systems correctly recognize your brand, what it associates you with, and where that understanding breaks down across your site and external mentions.

7. Improve Internal Linking With Contextual Signals

Internal links don’t just pass authority anymore - they also help reinforce what your content is actually about. In 2026, they play a bigger role in strengthening topical clarity.

To make internal linking more effective:

Use descriptive anchor text instead of generic phrases like “click here.”
Clearly signal what the linked page is about.
Connect related topics to strengthen semantic relationships.
Link naturally within relevant sections, not randomly.

For example, instead of writing:
Learn more here.

Write:

Learn more about our B2B SEO strategy framework.
This improves crawl efficiency, strengthens context, and helps AI systems better understand how your topics connect.

8. Optimize for Content Clarity and Summarizability

AI systems often summarize content before recommending it. If your page is hard to understand, AI systems may skip it when summarizing and recommending content. This happens because AI models prioritize pages that are clear, concise, and structured - essentially reflecting how LLMs choose and prioritize content.

To keep things simple:

Use short paragraphs that are easy to read
Avoid jargon or overly technical terms
Break up big blocks of text
Start with a quick, clear explanation before diving into details
Use headings to guide the reader

A simple test: can someone explain your page in three sentences? If not, simplify it. Clear content helps both people and AI quickly understand and trust your page.

9. Align On-Page Content With Off-Page Signals

Your website should tell the same story as your social media, listings, and press coverage. Mixed messages confuse AI tools and people about what your brand really stands for.
When your messaging lines up everywhere, AI understands your brand better, which boosts visibility in search and recommendations.

Tools like Track My Visibility can help you see:

Where AI mentions your brand
Which searches are your pages missing
What needs fixing on your site or elsewhere

Consistent messaging takes you from guessing whether your brand is being noticed to actually seeing how AI and search engines trust and reference you.

10. Monitor AI Visibility Alongside Rankings

In 2026, high rankings don’t tell the whole story. Your pages might rank well, but still get ignored by AI tools.

It’s important to track:

Whether AI mentions your brand
Which competitors appear in AI-generated answers
Which of your pages are cited - and which are skipped

Traditional SEO tools only show clicks and impressions, not recommendation visibility. Tools like Track My Visibility can also show how to measure your visibility in AI content across different AI platforms, helping you identify which pages are being recommended and which are being overlooked.

Monitor AI answers for your key queries
Highlight pages that aren’t showing up
Suggest actionable fixes for specific URLs

This helps SEO teams go beyond rankings and optimize for AI-driven discovery and recommendations.

The Evolution of On-Page SEO in 2026

On-page SEO has shifted from keyword engineering to clarity engineering.

Search engines now evaluate:

User satisfaction
Semantic relevance
Entity consistency
Technical structure

AI systems add another layer:

Confidence in summarization
Consistency across platforms
Verifiable credibility signals

The brands that succeed are those that make their content easy to understand, validate, and reference.

Final Thoughts

On-page SEO in 2026 requires a broader perspective.
You’re no longer optimizing only for ranking positions. You’re optimizing for inclusion in AI-generated answers.
By implementing these 10 practices - strengthening intent alignment, improving structure, defining your entity clearly, and monitoring AI visibility with platforms like Track My Visibility - you can ensure your pages perform across both traditional and AI-powered search environments.
Search visibility is no longer just about where you rank. It’s about whether you’re part of the answer.

Learn More: AI Search Analytics: Tracking Brand Visibility in AI Search

FAQs

1. What is on-page SEO in 2026?

On-page SEO today isn’t just about keywords. It’s about creating content that’s clear, easy to summarize, structured well, and understandable by AI systems, while still solving the searcher’s problem.

2. Why is AI visibility important for my website?

AI tools like ChatGPT and Perplexity often provide summarized answers instead of traditional search results. If your content isn’t easy for AI to interpret, it may not be cited, even if you rank on Google.

3. How can I make my content AI-friendly?

Lead with the main answer, use clear headings, descriptive titles, internal links, structured data, and consistent brand messaging. This helps both readers and AI understand and trust your content.

4. What role does structured data play in SEO?

Structured data, or schema markup, tells AI and search engines what your content is about - from products and services to FAQs and reviews. It doesn’t change the page visually but increases the chance of rich results and AI references.

5. How can I measure on-page SEO performance in 2026?

Look beyond rankings. Track both traditional metrics like clicks and impressions, and AI visibility - which pages are cited, which competitors appear, and where your content might be missing in AI-generated answers.