AEO Isn't SEO: Why You Should Let Bots Crawl Your Site in 2026

Ben Santora — Fri, 10 Jul 2026 10:40:36 +0000

For twenty years, robots.txt existed to keep bandwidth-hungry crawlers off your server. In 2026, for anyone who wants their work to actually be found, that instinct is backwards.

If your goal is showing up when someone asks Claude, ChatGPT, or Perplexity a question — not just ranking on page one of Google — you need AI crawlers hitting your pages. That's AEO: Answer Engine Optimization, and it runs on a completely different mechanism than classic SEO.

Here's what that actually means in practice, without the hand-waving.

The bots aren't one bot

The biggest mistake I see (and made myself, initially) is treating "AI crawlers" as a single monolith you can allow or block with one User-agent: * rule. They're not. Each provider runs multiple, functionally distinct bots, and blocking one does nothing to the others:

OpenAI: GPTBot (training), OAI-SearchBot (powers ChatGPT Search results), ChatGPT-User (live fetch when a user clicks a citation)
Anthropic: ClaudeBot (training), Claude-SearchBot (search indexing), Claude-User (fetches a page at a user's direct request)
Perplexity: PerplexityBot (its search index crawler), Perplexity-User (user-triggered fetches)
Google: Google-Extended (opts you in/out of Gemini training specifically — separate from Googlebot, which handles classic Search and isn't affected by this toggle)

Blocking only ClaudeBot does not block Claude-SearchBot or Claude-User — each token needs its own directive. The same logic applies across every vendor: training, search-indexing, and live-fetch are three separate decisions, not one.

A concrete consequence: GPTBot scrapes content to train OpenAI's models, while OAI-SearchBot indexes content specifically for ChatGPT Search results — so blocking GPTBot alone doesn't remove you from OpenAI's training pool if you left OAI-SearchBot open, and vice versa. You have to decide, per bot, which behavior you're actually opting in or out of.

robots.txt is a request, not a lock

Worth being blunt about: robots.txt was standardized as RFC 9309 in 2022, but it's a voluntary request that well-behaved crawlers honor — it carries no legal force and doesn't technically block access. Real enforcement, if you need it, happens at the server or CDN layer.

Most of the major players do comply. GPTBot, ClaudeBot, and PerplexityBot each identify themselves with a distinct user-agent string and obey robots.txt rules. But not everyone plays fair — Perplexity has been caught running undeclared crawlers that rotate user-agents, IPs, and ASNs specifically to evade no-crawl directives, and Cloudflare's own investigation concluded that robots.txt alone isn't a reliable defense against that behavior if a bot chooses to ignore it. If you actually need a hard block against a non-compliant crawler, that's a WAF/CDN-level rule, not a text file.

The canonical tag claim doesn't hold up

I want to flag something I said elsewhere that I'd retract: the idea that a rel="canonical" tag is "the key" signal telling AI crawlers "this is the source of truth." That's not documented anywhere I could verify. Canonical tags are a Google/Bing convention for deduping search index entries — there's no confirmed spec showing GPTBot, ClaudeBot, or PerplexityBot parse and weight it the same way for training ingestion or answer attribution. Treat that one as an unverified guess, not a fact to build a strategy around.

What actually moves the needle

Based on current documentation and crawler behavior:

Explicit per-bot robots.txt entries. List GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, and Google-Extended individually. A blanket Allow: / for * doesn't give you granular control over training vs. search vs. live-fetch.
Server-rendered, crawlable HTML. Not every AI crawler executes JavaScript. If your content only appears after client-side rendering, some bots simply won't see it.
Structured data (schema.org). This one has actual documented consumption by search and AI systems, unlike the canonical-tag claim above.
llms.txt. An emerging, not yet standardized convention (proposed by Answer.AI's Jeremy Howard and collaborators) that gives LLMs a curated map of your site's most important content, similar in spirit to sitemap.xml. Adoption sits at around 10% of domains as of mid-2026, so it's early — but it's low-effort and zero-risk to add.
Check your CDN layer separately from your robots.txt. This is the one people miss most. If you're on Cloudflare or a similar service, its bot-management dashboard can silently override what your actual robots.txt file says. Research on Cloudflare's network found a meaningful share of sites accidentally blocking major AI crawlers at the CDN while their robots.txt says "allow" — the two layers have to agree, or your intended policy does nothing.

A sane starting point

# Allow AI search / retrieval (what drives citations)
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /

# Training crawlers — allow if you want long-term model familiarity
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /

# Standard search engines, unchanged
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Adjust the training-bot section if you have genuinely proprietary content you don't want memorized — that's a legitimate reason to Disallow GPTBot/ClaudeBot/Google-Extended specifically while still keeping the search/retrieval bots open for citations. But for a personal site, a portfolio, or a blog whose entire point is being read — open the gate.

If you're auditing your own robots.txt, check your server logs for 403s against OAI-SearchBot, Claude-User, and PerplexityBot specifically — that's the fastest way to catch an accidental block costing you AI-search visibility.