Your robots.txt file does not block anyone. It is a text file full of polite requests, and the standard behind it says so in plain words. RFC 9309, the IETF specification for the Robots Exclusion Protocol, states that the rules "are not a form of access authorization."
That matters more than it used to. Your site is now read by AI crawlers that collect training data, search bots that decide whether you appear in chatbot answers, and fetchers that pull your pages when a user asks a question. If you want a say in any of that, it helps to know what robots.txt actually does, who is listening, and what to add when a request is not enough.
robots.txt is a convention, not a control
RFC 9309 formalized the Robots Exclusion Protocol in 2022. The spec says crawlers are requested to honor the rules in your file. Requested. Nothing puts the file between a crawler and your server: a cooperative crawler fetches robots.txt separately and decides for itself whether to comply, and nothing in the protocol can stop a request. Well-behaved crawlers comply because their operators choose to comply. Cloudflare's documentation puts it directly: robots.txt directives are voluntary, and crawlers can ignore them.
The AI labs split their crawling into separate bots
OpenAI documents three user agents. GPTBot crawls content that may be used to train its foundation models, and disallowing GPTBot tells OpenAI your content should not be used for that training. OAI-SearchBot is used to surface websites in ChatGPT search results; OpenAI says sites that opt out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links. ChatGPT-User fetches a page when a user asks ChatGPT something; because these requests are user-initiated, OpenAI notes that robots.txt rules may not apply. The settings are independent, and OpenAI says search can take about 24 hours to adjust to a robots.txt update.
Anthropic documents a similar split. ClaudeBot collects web content that could contribute to model training, Claude-SearchBot works on search result quality, and Claude-User handles fetches directed by a user's question. Anthropic says its bots honor robots.txt directives, supports the non-standard Crawl-delay extension, and asks you to set the rules on every subdomain you want covered. As of October 30, 2026, Anthropic has also begun recognizing a machine-readable signal, based on IETF draft work, that lets a site express a "no AI training" preference, with the rollout happening over time. A robots.txt rule that names ClaudeBot directly takes priority over that general signal.
The practical consequence: "allow AI" and "block AI" are not your only options. You can disallow the training crawlers while allowing the search crawlers, so your pages can still be cited in chatbot answers. That split is the part most site owners miss.
When a request needs teeth, enforce it on the request path
If you want certainty instead of cooperation, the block has to happen on the request path itself: a CDN or WAF rule, your origin server, or your app requiring a login. Cloudflare's AI Crawler and Agent access policies are the clearest example. You set an action for each behavior (Search, Agent, Training), and the Block action stops matching requests at Cloudflare's edge. Two details are worth knowing. The Training behavior also offers a Disallow action that only writes robots.txt directives, which stay voluntary unless you enable the option that additionally blocks AI training crawlers at the edge. And for new domains, all three behaviors default to Allow, so doing nothing is itself a choice.
You can do the same thing without Cloudflare: deny requests by user agent in your server configuration, or block the published IP ranges the labs maintain (OpenAI publishes a JSON list per bot). One caveat from Anthropic's documentation: blocking their crawler IPs can impede their ability to read your robots.txt, so they warn it may not work as a persistent opt-out on its own. My advice: keep robots.txt as the stated preference, and treat request-path rules as the enforcement layer, not a replacement for the file.
The file that does the opposite: llms.txt
Blocking is only half of the decision. The llms.txt proposal, first published in 2024 and updated to v2 in 2026, suggests a markdown file on your site that gives agents concise, expert-level information plus links to markdown versions of your pages. It is a proposal, not a standard, but it notes that thousands of sites now publish one, and OpenAI, Anthropic and Gemini publish their own. If you want AI assistants to describe your product accurately, this is the constructive counterpart to robots.txt: one file states what stays out, the other helps what comes in get read correctly.
How I would approach it
This part is advice, not documentation. Ask three questions. Do I want my content in training datasets? Do I want it in AI search answers? Do I want user-triggered fetches to work? Then write one robots.txt group per bot to match your answers. If "no" needs to mean "no," put a real block behind it, whether that is a CDN rule, a server rule or a login wall. If you want to be read well, consider an llms.txt. None of it is permanent; you can change the file tomorrow.
What this adds up to
robots.txt is where you state your preference, and the standard itself says it is not access authorization. The major AI labs document separate bots for training, search and user-triggered fetches, so you can choose per purpose. Only a control on the request path, a CDN rule, your server configuration or a login, can actually stop a request. And llms.txt exists for the opposite goal: helping agents read what you want them to read.
The next step is small. Open your robots.txt, decide what it should say about each bot, and decide whether your "no" needs enforcement behind it.
Source pages: RFC 9309: Robots Exclusion Protocol, OpenAI crawlers, Anthropic's crawler documentation, Cloudflare AI Crawler and Agent access policies, The llms.txt proposal.
I write about startups and AI at startupwiki.substack.com.
Cover photo: Phil Hearing on Unsplash.

Top comments (0)