robots.txt AI training opt out: one token, four companies

Originally published on proxymint.

One robots.txt AI training token asks OpenAI, Google, Microsoft and Anthropic not to train on your site. How to add it, when each honours it, how to test it.

TL;DR

• Add two lines, User-agent: __usage_ai-training and Disallow: /, and the opt-out is in place. OpenAI, Google, Microsoft and Anthropic said on 30 October 2026 they'll honour that group, which saves a separate rule for every training bot.
• Don't expect it to work everywhere yet. Anthropic says it has begun recognising the signal. Microsoft starts on 1 February 2027, while OpenAI talks about the coming months and Google aims for 2027.
• Check your GPTBot, ClaudeBot and Google-Extended lines against the token, because they still count and can overrule it. A rule that names ClaudeBot wins at Anthropic and an explicit Google-Extended rule wins at Google, so the two have to agree.
• Put Allow lines first, since the draft's own example trips Python's parser. Given Disallow: / above Allow: /press/, Python 3.9.6's robotparser refused /press/kit too.

Key takeaways

• Add the token group today, even before everyone reads it. Old parsers skip it like any unknown bot, so two lines is the whole cost, and it already covers companies that adopt it later.
• Read your per-bot groups next to it. A Google-Extended allow tells Google yes whatever the token says, and at Google that one rule covers Gemini training and grounding together.
• Order carve-outs Allow first and the parser stops mattering. First-match parsers like Python 3.9.6's then give the same answer as RFC 9309 longest-match parsers.
• Crawler builders have to look the token up themselves. It's a policy name, not a crawler name, and a stock parser asked about GPTBot or ClaudeBot falls through to the * group.

Read the full guide on proxymint, with the commands, tables and sources.