OpenAI to honor a shared robots.txt signal for AI training preferences across providers

#ai #openai #web

📅 October 30, 2026 • Source: openai.com

OpenAI has announced that it will honor a shared signal in robots.txt based on the draft developed by the AI Preferences working group at the IETF. The announcement was published on October 30, 2026. The change targets a known friction point in web publishing: a site owner who wants to state preferences about AI training on their content currently writes separate rules for every crawler, one block per provider, and repeats the work each time a new crawler appears. Under the approach described by OpenAI, the owner declares those preferences once, and several participating AI providers read the same single signal. The company says the rollout will begin in the coming months and describes the initiative as voluntary, inviting other providers to support the same signal.

How the shared signal works
The signal is placed in robots.txt, the file site owners already use to manage automated access. Its syntax follows the draft developed by the IETF working group on AI Preferences, known as aipref. The practical difference is in the number of rules: instead of maintaining one set of directives per crawler, the owner expresses a training preference a single time, and the declaration applies across several providers that honor it. OpenAI states that its existing crawler controls remain in place and that the new signal does not replace them, so current robots.txt configurations keep working during the transition. Participation is voluntary on both sides: a site owner chooses whether to add the signal, and each AI provider chooses whether to honor it. OpenAI frames the announcement as an invitation to the industry and asks other providers to adopt the same signal, which would extend the reach of one declaration without extra work for the site owner.

IMPACT
For site owners and publishing teams, the immediate change is maintenance: one shared declaration can replace repeated per-crawler entries as participating providers switch on support in the coming months. Web and SEO teams can review their robots.txt now, decide what the training preference should say, and verify that existing per-crawler directives still apply, since OpenAI confirms both mechanisms coexist. For compliance and data governance functions, the signal adds a documented, machine-readable expression of preferences that can be referenced in internal policies on content use. The voluntary design carries a practical limitation: coverage depends on each provider adopting the signal, so a declaration does not by itself bind providers outside the initiative, and per-crawler rules stay relevant for them. During the rollout window, some providers may honor the shared signal while others do not, which argues for keeping both mechanisms in place until adoption is broad.

The announcement is available on openai.com, and the working group documents are tracked on ietf.org.

Sources: openai.com and ietf.org