OpenAI, Google, Microsoft and Anthropic to support a unified AI training opt-out token in robots.txt

#news #openai #seo #ai

📅 2026-10-31 • Источник
SiliconANGLE

OpenAI, Google, Microsoft and Anthropic to support a unified AI training opt-out token in robots.txt

Four of the largest AI companies have agreed to honor one common machine-readable signal that lets website publishers restrict the use of their content for AI training. OpenAI, Google, Microsoft and Anthropic will support a single shared token in robots.txt, the standard file that site operators already use to publish rules for automated crawlers. Once each company completes its rollout, a publisher will be able to tell all four companies’ crawlers not to use its content for training with one line of text, subject to each crawler honoring the voluntary signal, instead of separate opt-out rules for every crawler.

The specification

The shared token is __usage_ai-training. It is defined in an individual draft at the IETF titled Machine-Readable Training Preference Signal. The draft is authored by Kevin Kelley of Anthropic, is dated October 30, 2026, and expires on May 3, 2027. An individual draft is an early stage of IETF standardization — the proposal is documented and open for review but has not been adopted as a formal standard.

Rollout timelines

The rollout picture comes from press reports and company blogs rather than a signed standard. OpenAI confirmed on its blog that support will arrive in the coming months. According to SiliconANGLE’s report, Anthropic has already begun honoring the token, Microsoft is reported to start on February 1, 2027, and Google is reported to implement support during 2027. Those timelines were not found in the companies’ own primary materials at the time of writing. Publishers who add the token now will see uneven enforcement across crawlers during a transition period that could stretch through most of 2027.

Limits to note

The signal builds on existing practice: robots.txt has been used for decades as a voluntary convention between site operators and crawler operators, and major AI companies already publish instructions for their crawlers. The token expresses a preference and relies on voluntary compliance by the crawler operator, so it does not create an enforcement mechanism by itself. It covers training use specifically, while search indexing or real-time retrieval follow separate rules unless a site restricts those as well. Legal consequences for a violation depend on applicable law in each jurisdiction, not on the token. OpenAI’s own announcement adds that the new signal will not replace the controls it already provides, so existing robots.txt rules stay in place and the token does not guarantee exclusion from training.

What to do

Adding the token is a one-line change in an existing file, but support arrives on different dates per provider, so verification matters. Site owners who care about training use can add the signal now and check crawler behavior against each company's stated rollout date as it passes.

Sources
siliconangle.com — https://siliconangle.com/2026/10/30/google-anthropic-and-others-to-support-new-ai-crawling-opt-out-standard
openai.com — https://openai.com/index/helping-websites-express-ai-training-preferences/
blogs.bing.com — https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/
datracker.ietf.org — https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/