Google, Microsoft and OpenAI have signaled support for a shared way for websites to express AI-training preferences in robots.txt. The proposed approach could reduce a growing operational problem for publishers: managing separate crawler directives for multiple AI providers while trying to preserve visibility in conventional search and AI-generated answers.
The plans are not an immediate, universal switch. Google says it plans to support the shared control in 2027, Microsoft has published a staged transition for Bing, and OpenAI says it is willing to honor a common signal in the coming months. Still, the announcements establish an important principle: AI-training preferences may increasingly be handled through a common standard rather than a collection of provider-specific rules.
What the shared AI-training signal is intended to do
The technical basis is an Internet Engineering Task Force draft, draft-kelley-aipref-training-preference-signal-00, published on October 30, 2026. It describes a policy-token approach for expressing AI-training preferences through robots.txt.
This matters because robots.txt is already a familiar part of technical SEO operations. Site owners use it to communicate crawler access preferences, but AI systems have introduced more nuanced questions. A publisher may want to limit use of content for model training while still appearing in search results, or may want different rules for training and for using pages to ground an AI-generated response.
The proposed shared signal is designed specifically around the training question. It does not mean every existing AI crawler directive disappears, and it does not automatically settle every question about crawling, indexing or citation in AI answers.
Google's planned interpretation has an important detail
In its Google Search Central announcement on AI training control, Google said it plans to support the shared control in 2027. The company also described a consequential default for sites using the shared token.
If a website uses the shared AI-training control but does not explicitly state a preference for Google-Extended, Google plans to treat that configuration as if Google-Extended had been disallowed. Google presented this as alignment with the broader intent of simplifying publisher controls.
For technical teams, that detail means a future implementation should not be treated as a routine robots.txt addition. The interaction between a shared preference and existing Google-specific directives will need a deliberate review. Publishers that currently rely on Google-Extended settings should document their present policy before changing anything.
Google's announcement concerns the planned support and interpretation of the preference. It should not be read as proof that a particular configuration will affect rankings, indexing, traffic or inclusion in every AI answer.
Bing separates AI training from AI grounding
Microsoft's Bing announcement takes a more granular approach. It outlines independent preferences for AI Training and AI Grounding, plus page-level overrides.
That distinction is particularly relevant for publishers. AI training concerns whether content can be used in developing models. AI grounding concerns whether content may be used to support or inform an AI response. Those uses can have different editorial, commercial and discoverability implications.
Microsoft's transition begins February 1, 2027. Bing also describes a grounding transition beginning in December 2026 and says temporary grounding options expire on September 30, 2027. The staged timetable gives publishers time to understand the new model, but it also means directives may operate differently during the transition.
OpenAI has likewise said it is willing to honor a common robots.txt signal for AI-training preferences. Its statement on helping websites express AI training preferences says the shared signal would be implemented in the coming months and would not replace existing crawler controls.
| Organization | Announced direction | Timing stated in the research |
|---|---|---|
| Support a shared AI-training control in robots.txt, with a planned Google-Extended interpretation when no explicit preference is supplied. | Planned for 2027 | |
| Microsoft Bing | Separate AI Training and AI Grounding preferences, including page-level overrides. | Grounding transition from December 2026; transition begins February 1, 2027 |
| OpenAI | Willingness to honor a common AI-training preference without replacing existing crawler controls. | Implementation in coming months |
What publishers should do before rollout
There is no reason to rush a production change before the relevant implementation details and rollout dates apply. There is, however, a strong case for preparation.
First, inventory current robots.txt directives, including any AI crawler controls and Google-Extended preferences. Many organizations have accumulated directives through separate legal, editorial and SEO decisions. A current inventory makes it easier to identify conflicts or unintended defaults when shared controls arrive.
Second, separate policy decisions from implementation decisions. Stakeholders should decide whether their policy differs for AI training, AI grounding, search crawling and page-level exceptions. Bing's model makes clear that a simple yes-or-no policy may not reflect the choices available to publishers.
Third, ensure changes are version-controlled and tested. A malformed or overly broad robots.txt rule can affect crawling beyond the intended AI use case. Teams should record the rationale, implementation date and affected user agents or tokens, then monitor crawl behavior and organic discovery after deployment.
Finally, do not assume an AI-training preference equals an AI-visibility strategy. Controls express a preference about a defined use of content. They do not by themselves make a page more citable, improve content quality, or guarantee whether a system will surface a brand in an answer.
For teams reviewing how crawler settings, content accessibility and AI discovery fit together, create a GEO Search Leads account to begin assessing your website's GEO readiness and identify optimization opportunities. That can help turn a one-off robots.txt policy review into a broader, documented approach to the pages and content signals that support discoverability.
Other measurement changes can complicate diagnosis
The shared-control announcements arrived alongside two separate developments that reinforce the need for careful interpretation of SEO reporting.
SE Ranking's study of unrelated Google results reported a surge in “poisoned” SERPs, where results appeared unrelated because of one-word fragments. The company found that 57.3% of tracked keywords were affected overnight, rising to 67.2% within a week. Most of the reported issues began outside the top eight results.
Those are findings from SE Ranking's tracked keyword set, not a measurement of all Google searches. Even so, they are a reminder that abrupt ranking changes in an automated dashboard can reflect an anomalous results set rather than a genuine loss of relevance or visibility. Investigate the live SERP and the query interpretation before treating a movement as a site-wide performance decline.
PageSpeed Insights also adopted Lighthouse 13.6.0, released October 27, 2026. The update changes the Lantern simulation used for lab scores, so a score can move even if the underlying site has not changed. When comparing performance reports, use the same Lighthouse version where possible and give appropriate weight to field data. Lab scores remain useful diagnostics, but version changes can affect trend lines.
Frequently Asked Questions
Does the shared robots.txt signal block AI search visibility?
Not necessarily. The shared proposal concerns AI-training preferences. Microsoft separately distinguishes AI grounding from training, and OpenAI says the common training signal would not replace existing crawler controls. The visibility effect of any configuration depends on the provider's documented controls and the specific use being addressed.
When will Google support the shared AI-training control?
Google's October 30, 2026 announcement says support is planned for 2027. The announcement does not establish that the control is already active for all sites.
What happens if a site uses the shared signal but does not specify Google-Extended?
Google says it plans to treat that situation as though Google-Extended had been disallowed. Site owners using the shared control should review their existing Google-specific preferences before implementation.
Why might rank-tracking reports show a sudden drop that is not a real loss?
SE Ranking reported unrelated-result anomalies across its tracked keywords, which can distort automated position measurements. Check the actual search results and query interpretation before concluding that a ranking decline reflects a change to the site.
Conclusion
A common AI-training preference could make publisher controls easier to manage, but it will not eliminate the need for precise policy and technical review. With Google, Bing and OpenAI moving on different timelines and Bing separating grounding from training, the practical priority is to document current directives, understand the choices each platform offers and avoid reading dashboard shifts as proof of a visibility change without validation.
Top comments (0)