Google, Microsoft, OpenAI and Anthropic Just Agreed on One robots.txt Signal for AI Training — Explained Honestly

On 30 October 2026, four of the biggest names in AI — Google, Microsoft, OpenAI and Anthropic — lined up behind a single shared robots.txt signal for AI training preferences. For years, telling AI companies not to train on your website meant maintaining a growing list of crawler names, one vendor at a time. The new idea is simple: write one preference, in one place, and every participating provider reads it the same way.

That is genuinely good news for publishers. It is not, however, a switch that turns AI training off everywhere tomorrow. The timelines are uneven, the specification is still a draft, and a robots.txt preference is not a security lock. This article says what actually changed, what each provider actually committed to, and what did not change — plainly, without the hype.

Nothing here is sponsored — no affiliate links on this page. Every fact below comes from reporting of the 30 October 2026 announcements (Unite.AI, MonsterMegs, Blockchain.News) and the IETF Internet-Draft they describe. I have not tested any provider's implementation of the new signal, and where a detail comes from secondary reporting rather than a provider's own announcement, I label it as reported.

What actually changed

One signal instead of one rule per provider. Until now, a site owner who wanted to opt out of AI training had to identify each provider's crawlers and maintain separate rules for each — GPTBot for OpenAI, Google-Extended for Google, and so on, with new names appearing as new companies and products arrived. The shared signal replaces that patchwork with a single policy token that participating providers all agree to read the same way.

The announcement was coordinated, not solo. Google announced the shared control on its Search Central blog (John Mueller, Google Search Advocate), saying it "will help websites express their AI training preferences across multiple AI providers in a single step." OpenAI announced the same day that it will honor the shared signal. Anthropic authored the underlying draft, and Microsoft signalled support the same day. That coordination is the real news: the companies behind most commercial model training have publicly agreed on the shape of one control, which was not true a week ago.

It is based on IETF work, not a private deal. The signal is defined in an Internet-Draft from the IETF's AI Preferences (aipref) working group, titled "Machine-Readable Training Preference Signal" (draft-kelley-aipref-training-preference-signal-00), authored by Kevin Kelley of Anthropic, dated 30 October 2026 and expiring 3 May 2027. Google describes the shared control as based on that working group's work, and OpenAI framed its move as drawing on the same working-group effort.

How the signal works

The mechanism is deliberately boring — and that is its strength. It introduces a single policy token, __usage_ai-training, that sits in the User-Agent line of an ordinary robots.txt group. There are no new directives and no new grammar: the path language, longest-match precedence and group structure of the robots.txt standard (RFC 9309) are unchanged.

The draft's own example group expresses a preference against AI training for all paths except those under /press-kit/:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Disallow expresses a preference against AI training for matching paths; Allow expresses a preference in favour. Because it reuses existing robots.txt path rules, a preference can be set per directory rather than site-wide — useful if you are happy for product pages to be summarised by assistants but want premium guides, course material or member content left out of training sets.

Three compatibility details matter:

• Parsers that do not recognise the token simply ignore the group, the same way they ignore rules for any bot they have never heard of. Adding the group to an existing file should not change how search engines or other crawlers read the rest of it.
• Wildcard groups do not count. The draft says a User-Agent: * group "must not be interpreted as matching the policy token." If your robots.txt has no __usage_ai-training group, no training preference has been expressed at all, whatever your catch-all rules say.
• Specific rules win. The draft lets providers treat their own tokens, such as Google-Extended, as more specific than the shared one. Google confirmed it will work that way: sites that explicitly allow Google-Extended remain eligible for grounding citations in Gemini Apps, even if they opt out broadly with the shared signal.

What each provider actually committed — and when

The timelines are uneven, and that unevenness is the most important honest detail in this story. There is no single switch-on date.

Google — targeting support in 2027. Google says it aims to support the new AI training control in 2027. When a site uses the shared token to opt out and does not explicitly mention Google-Extended, Google plans to interpret that as if the publisher had disallowed Google-Extended — the training control Google has offered since 2023, which is not going away. Company-specific controls, Google says, "will continue to function alongside this new joint signal." Google also ties the move to regulation: the proposal is consistent with the EU Code of Practice for General Purpose AI model providers under the EU AI Act, which recognises robots.txt as a primary way for publishers to state machine-readable preferences. And the search reassurance holds: Google-Extended has long been said not to affect inclusion or ranking in Google Search, because Googlebot crawling for Search uses a separate product token — so opting out of training should leave organic visibility untouched.

OpenAI — "in the coming months." OpenAI announced on 30 October 2026 that it will honor the shared signal, describing the effort as voluntary and aimed at removing the burden of maintaining separate rules per provider. It says it aims to implement the signal in the coming months and will share another update with implementation details when it does. No firmer date has been given.

Anthropic — authored the draft; implementation reported, not confirmed here. Anthropic's Kevin Kelley authored the draft, and same-day reporting says Anthropic has already started implementing support. I label that as reported, because I have not verified it against an Anthropic announcement in the sources fetched for this article.

Microsoft — building on the shared preference, with a separate grounding setting (as reported). Reporting says Microsoft will build on the shared preference while adding a separate setting for AI grounding, and that publishers should not have to choose between Bing discoverability and training choice. Again, that is reported detail, not a Microsoft announcement I verified directly — and the distinction matters: grounding (an assistant citing your pages) is not the same thing as training (your pages changing a model's weights), and a publisher may reasonably want different answers for each.

What did NOT change

Your existing controls still work and are not replaced. OpenAI says the new signal will not replace the controls it already provides through robots.txt, which will continue to apply and be respected. Google says the same about Google-Extended. Do not delete your existing GPTBot or Google-Extended rules on the strength of today's announcement — they are the controls that actually work today, while the shared signal is still rolling out.

Search and training stay separate. This is the distinction most people will get wrong. Each provider's settings are independent: a webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot for training. Allowing a search bot can preserve search eligibility while disallowing training — and, on Google's side, opting out of training should not touch your search ranking. Keep those two questions separate when you read your analytics.

A training preference does not stop crawling. The draft states plainly that the policy token "is not a directive for crawling behavior." Content-usage preferences and crawler traffic-management signals are independent. Setting the shared token tells a provider how you want your content used; it does not, by itself, stop that provider's bots from fetching your pages. If AI crawler traffic is eating your server resources today, the shared token will not fix it — you still need user-agent groups for specific crawlers, or blocking and rate limiting at the server or CDN level.

It does not remove what was already collected. Updating robots.txt does not show that previously collected material has been removed from datasets or existing models. The signal governs future training preferences, as read by cooperating providers. It is not a deletion request and not a retroactive cleanup.

It is voluntary, and robots.txt is not enforcement. Participation is voluntary on both sides. Robots.txt is a preference signal, not access control or a security measure: a non-cooperating scraper can simply ignore it. The vocabulary draft behind the token says the same thing in its own words — it does not ensure preferences are followed and leaves the choice of whether to follow them to the recipient.

Who this helps — and who should wait

• Publishers, bloggers and small site owners tired of the patchwork: you are the intended audience. One group, one preference, read the same way by participating providers — once they implement it. That is a real reduction in busywork, even if the payoff arrives through 2027 rather than this week.
• Sites that want per-section control: the path rules let you opt out broadly and carve out exceptions (a press kit, public documentation) or vice versa. That granularity is genuinely useful, and it is already specified in the draft even before enforcement lands.
• Anyone hoping this fixes crawler load or removes old training data: wait, and do not confuse this signal with either problem. Crawler load needs server/CDN controls; old data needs a different conversation entirely, with the provider in question.
• Anyone who wants a guarantee today: wait. Until Google's 2027 support, OpenAI's "coming months" implementation and the other providers' details arrive, the shared signal is a published preference format with public backing — not an enforced control you can rely on everywhere.

What to do now (practical, hedged)

1. Read the draft before you rewrite anything. The specification is public and short, and it expires on 3 May 2027 unless revised — so expect details to change before providers enforce it. You do not need to rewrite your robots.txt today.
2. Leave your existing controls in place. GPTBot, Google-Extended and your other provider rules are what actually work right now. The shared signal is an addition for the future, not a replacement you should switch to by deleting the old rules.
3. Audit what you already have. Confirm Googlebot is not accidentally blocked, note your Google-Extended and GPTBot rules, and remember that an explicit Google-Extended rule will override the shared signal for Google. Adding a __usage_ai-training group now is low risk — parsers that do not recognise it ignore it — but treat it as an early preference statement, not as protection that is already enforced.
4. Decide what you actually want, section by section. Opting out of training, allowing it, and allowing it only for some sections are all reasonable positions. The right answer depends on whether AI assistants send you customers or simply summarise your work without a click — and grounding visibility is a separate question from training.
5. Treat crawler load separately. If your logs show AI bots hammering your site, rate-limit or block at the server or CDN layer. A training preference will not slow them down.

What we cannot verify — stated plainly

• Anthropic's and Microsoft's implementation status: reported by same-day coverage, not verified here against their own announcements. Their commitments are described above as reported, not as confirmed facts.
• Any single switch-on date: there is none. Google targets 2027, OpenAI says "in the coming months," and the draft itself expires in May 2027 unless revised. Do not trust any article that gives you one date for all providers.
• Whether the draft syntax is final: it is an individual Internet-Draft with no formal IETF endorsement or standing in the standards process. Google itself describes its commitment as a draft proposal. The syntax may change before enforcement.
• Enforcement against non-cooperating scrapers: there is none in this mechanism. Robots.txt expresses a preference; it does not compel compliance, and it does not remove previously collected material.
• Exact provider implementation details: OpenAI says it will share details when it implements the signal; Google's mapping to Google-Extended is the most detailed commitment published so far. Everything else is still to come.

The honest summary is this: for the first time, the largest model builders have publicly agreed on one way for publishers to say "do not train on this" — and the mechanism is elegantly simple, reusing a file every site owner already has. But agreement on a format is not enforcement, the rollout stretches into 2027, and the controls that protect you today are still the old per-provider ones. Keep those, learn the new signal, and judge the providers by what they actually implement — not by today's headlines.

---

Sources: Unite.AI, "OpenAI to Honor Shared Robots.txt Signal for AI Training Preferences", 30 October 2026, https://www.unite.ai/openai-to-honor-shared-robots-txt-signal-for-ai-training-preferences/ ; MLex, "Google, Microsoft, OpenAI, Anthropic to offer streamlined AI training tool", 30 October 2026, https://www.mlex.com/mlex/articles/2544745/google-microsoft-openai-anthropic-to-offer-streamlined-ai-training-tool ; MonsterMegs, "What The Shared AI Training Control Means", 30 October 2026, https://monstermegs.com/blog/ai-training-control/ ; Blockchain.News, "OpenAI to honor shared robots.txt AI training signal", 30 October 2026, https://blockchain.news/news/openai-honor-shared-robots-txt-ai-training-signal ; Daily AI Brief, "OpenAI shared robots.txt AI training signal", 30 October 2026, https://dailyaibrief.com/news/openai-shared-robots-txt-ai-training-signal-uQrvMeDk .

Full version with FAQ: https://toolscout-e2i.pages.dev/posts/shared-robots-txt-ai-training-control.html