robots.txt for AI crawlers: block training, keep AI search, and test that it works

At some point someone on your team will ask you to "block the AI bots". Before you add a blanket rule, it is worth knowing that the major AI providers run different crawlers for different purposes, and that blocking the wrong one removes your site from AI search answers you probably want to appear in.

This post is the developer version of a longer guide on the Crawl Foundry blog: what each bot does, how to express the common policies in robots.txt (including in Next.js), one grouping rule that trips people up, and how to verify the result.

Three purposes, three kinds of bots

AI companies fetch pages for three reasons:

• Training: collecting content to train future models
• Search: building the index behind the assistant's search feature, which decides whether you can be cited as a source
• User-initiated fetches: an assistant opening a specific page because a user asked it to

The main user agents, per the providers' own documentation:

• OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search), ChatGPT-User (user-initiated; OpenAI says robots.txt may not apply to these fetches)
• Anthropic: ClaudeBot (model development), Claude-SearchBot (search), Claude-User (user-initiated)
• Perplexity: PerplexityBot (its search index), Perplexity-User (user-initiated)
• Google: Google-Extended is not a crawler but a robots.txt token. It controls whether content Google already crawls may be used for Gemini training and grounding. It does not affect Google Search, and AI Overviews and AI Mode run on the regular Googlebot.
• Common Crawl: CCBot, whose datasets many model providers use

The list changes. Check the providers' documentation before a larger change.

The most common policy: block training, allow AI search

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /admin/
Disallow: /api/

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Several User-agent lines in a row form one group, so this is shorter than repeating the rules per bot.

The grouping gotcha

Notice that the Disallow: /admin/ and Disallow: /api/ lines appear twice. That is deliberate.

Under the robots.txt standard (RFC 9309), a crawler picks the group that matches its user agent most specifically and follows only that group. Rules are not merged with the * group. So the moment you give OAI-SearchBot its own group with Allow: /, it ignores everything you wrote under User-agent: *. If your private paths are only disallowed in the wildcard group, the bots you named explicitly are now allowed into them.

Any path restriction that should apply to every crawler has to be repeated in every named group.

The same policy in Next.js

If you use the App Router, app/robots.ts generates the file, and userAgent accepts an array:

import type { MetadataRoute } from "next";

const privatePaths = ["/admin/", "/api/"];

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: ["GPTBot", "ClaudeBot", "Google-Extended", "CCBot"],
        disallow: "/",
      },
      {
        userAgent: ["OAI-SearchBot", "Claude-SearchBot", "PerplexityBot"],
        allow: "/",
        disallow: privatePaths,
      },
      {
        userAgent: "*",
        allow: "/",
        disallow: privatePaths,
      },
    ],
    sitemap: "https://example.com/sitemap.xml",
  };
}

Keeping privatePaths in one constant makes the grouping rule hard to forget.

What robots.txt can't do

robots.txt is a request. Reputable crawlers follow it; nothing forces the rest to. If content must not be fetched, protect it on the server or behind a login.

Changes also only apply going forward and not instantly. Content that was already collected doesn't disappear, and OpenAI notes that it can take about 24 hours for OAI-SearchBot to pick up a change.

For Google Search specifically, robots.txt is the wrong tool for controlling what appears in snippets or AI Overviews. Page-level directives such as nosnippet or max-snippet do that. Blocking Googlebot removes the page from search entirely.

Verifying the result

1. Check the rules per bot. Parse your live robots.txt and ask, for each user agent you care about, whether a sample URL is allowed. Crawl Foundry has a free robots.txt checker for AI crawlers that shows which AI and search crawlers are allowed or blocked and which rule applies.

2. Check your edge. Many CDNs and WAFs ship their own "block AI bots" switches that act before robots.txt is ever read. A quick test with a spoofed user agent tells you whether the edge returns something other than your page:

for ua in "GPTBot" "OAI-SearchBot" "ClaudeBot" "Claude-SearchBot" "PerplexityBot"; do
  printf "%-18s " "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; $ua)" https://example.com/
done

A 403 or a challenge page for a bot your robots.txt allows means the CDN setting wins, whatever the file says. Spoofed requests don't come from the providers' IP ranges, so some bot-management products treat them differently from the real crawler; use this as a smoke test, not proof.

3. Check your logs. Server or CDN logs show which bots actually visit and what status they get. OpenAI and Anthropic publish the IP ranges of their crawlers, which lets you separate real visits from spoofed user agents.

One more thing: can they read the page?

Allowing a crawler doesn't help if it receives an empty shell. Many AI crawlers don't execute JavaScript, so content that only exists after client-side rendering may be invisible to them. Check the raw HTML response of your important pages, not the DOM in your browser.

The full guide, including setups for shops, software vendors, publishers and paywalled content, plus a list of common mistakes, is on the Crawl Foundry blog: How to Block or Allow AI Crawlers: GPTBot, ClaudeBot and Others in robots.txt. For the basics of the file itself, see the glossary entry on robots.txt.

Comments

[Dmitry Sergeev]

We need to produce a casual YouTube comment, short, one sentence max two. Must start with a specific reaction or question about this video, not generic praise. Should be casual, lower case start, maybe include "lol", "tbh", etc. No marketing, no URLs. No double hyphen. No fancy punctuation. Must be exactly the comment text (no quotes). Should be about robots.txt for AI crawlers: block training, keep AI search, test that it works. Could ask about how to verify that AI