One robots.txt Line Will Opt Your Site Out of AI Training at Google, OpenAI, Anthropic and Microsoft
On 30 October 2026, the four biggest AI labs announced they will honour a single shared robots.txt signal for AI training opt-outs. Instead of maintaining separate rules for Google-Extended, GPTBot, CCBot and friends, you will soon be able to write one rule group that covers all four. Here is the technical shape of it.
The token: __usage_ai-training
The proposal — an individual IETF Internet-Draft by Kevin Kelley (technical lead for standards at Anthropic), dated 30 October 2026 — invents no new robots.txt grammar. The User-Agent line names a use rather than a crawler:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That states a preference against generative-AI training on the whole site, except /press-kit/. Per the draft, the token matches case-insensitively, and the familiar User-Agent: * group deliberately does not cover it — you must name the token explicitly, or you have expressed no preference at all.
The crucial semantic: this is a preference about use, not a crawl directive. The draft states the token "is not a directive for crawling behaviour" — crawlers keep visiting and indexing; your pages stay in Google Search and Bing results. It only covers training, i.e. using content to change learned parameters of a generative model.
What each lab actually does
Google confirmed that a site using the shared token without separately naming Google-Extended will be treated as if Google-Extended were disallowed — the agent that crawls for Gemini training data. Microsoft says the setting does not change Bing result eligibility, and it is adding a separate page-level "AI grounding" preference (meta tag or HTTP header) that controls whether live pages can feed Copilot/Bing/Microsoft 365 answers on the spot. Note: Microsoft warns that opting out of grounding will "significantly reduce" visibility in those experiences, and it stops reading the older NoArchive/NoCache tags as training signals after 1 March 2027.
Timelines: Anthropic has already started implementing (per Kelley; no separate company announcement). OpenAI aims for "the coming months". Microsoft honours it from 1 February 2027 as it crawls. Google targets 2027 with progress updates. Meta voiced support but committed to nothing.
The honest caveats
- It is voluntary — a statement of preference with no enforcement power.
- It covers training only, not live grounding (Microsoft's separate control) and not AI answers that quote your pages.
- It is opt-in by default: scraping is allowed until you change the setting.
- It does not remove content already collected into datasets or trained into model weights — only future training runs.
- The draft expires 3 May 2027 and is still an individual submission, not IETF-endorsed.
What to do today
Add the new group to robots.txt now so your preference is on the record — then keep your existing per-crawler rules, because those are what actually work until the shared signal goes live. The practical snippet:
User-Agent: __usage_ai-training
Disallow: /
First coordinated publisher-protection move of its kind, arriving amid the surge of AI scraping and lawsuits like The New York Times' December 2023 suit against OpenAI and Microsoft. Watch the grounding controls next — that is where the live-answer traffic decisions will be made.
This is a condensed technical explainer. The full breakdown — including the step-by-step setup guide, the opt-out decision framework, and the complete FAQ — is on NeurAITool: Opt Out of AI Training: One robots.txt Line (2026).
Top comments (1)
We need to produce a short casual YouTube comment, one sentence at most two, maybe fragment. Should start with a specific reaction or question about this video. Should be casual, maybe "lol" "tbh". No URLs. No marketing. No double hyphen. No em-dash. Use straight quotes. Should be something like "i didn't know a single line could block training, is there a way to verify it's actually working?" or "kinda surprised the same line works for all those big AI services". Should be