DEV Community

Cover image for One robots.txt Rule to Opt Out of AI Training: What Developers Need to Know (2027)
Hafiz Waleed Ahmed
Hafiz Waleed Ahmed

Posted on

One robots.txt Rule to Opt Out of AI Training: What Developers Need to Know (2027)

#seo #webdev #ai #robots

On October 30, 2026, Google announced plans to support a shared AI training control in robots.txt, based on work from the IETF's AIPREF working group. Coverage the same day reported that Microsoft and OpenAI will honour it too, and that Anthropic has started building support.

If you build or maintain websites, here's what it means and what to do (all free).

The rule 

User-Agent: __usage_ai-training
Disallow: /
Enter fullscreen mode Exit fullscreen mode

Instead of a crawler name, the User-Agent line holds a policy token. Normal robots.txt path matching applies, so you can opt out site-wide and still allow a folder:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
Enter fullscreen mode Exit fullscreen mode

Timeline (it's not live yet)

  • Microsoft: from February 1, 2027
  • Google: "early 2027" per its crawler documentation note; the blog says 2027
  • OpenAI: "in the coming months"
  • Anthropic: implementation started, no date

It's an IETF draft, so the exact token or details could still change.

The Google-Extended catch

Google says that if you use the shared token to opt out and don't explicitly specify Google-Extended, it will treat that as if you disallowed Google-Extended. Google also says sites that allow Google-Extended are eligible to appear and receive clicks via grounding in Gemini Apps.

So if you want to opt out broadly but still be cited in Gemini, you'd add an explicit allow, knowing that it also lets Google use the content for Gemini:

# Opt out of AI training with all participating AI providers
User-Agent: __usage_ai-training
Disallow: /

# Optional: still allow Google's Gemini (training + citations in Gemini Apps)
User-Agent: Google-Extended
Allow: /
Enter fullscreen mode Exit fullscreen mode

Checklist

  1. Open yoursite.com/robots.txt and see what's already there.
  2. Decide: protect your content from training, or stay visible in AI answers?
  3. If opting out, add the shared token group.
  4. Decide on Google-Extended explicitly, so nothing happens by default.
  5. Google Search rankings aren't affected. Re-check the spec in early 2027.

What would you choose for your projects: block, allow, or a mix by folder?

Top comments (2)

Collapse
 
alexshev profile image
Alex Shev •

The important distinction is between expressing a preference and proving it was honored. A deployment checklist should say which crawler is addressed, how the rule is served and cached, and what evidence the site owner can retain if a policy dispute arises.

Collapse
 
citedy profile image
Dmitry Sergeev •

We need to produce a short YouTube comment. Must be casual, start with lowercase, not formal, no marketing. Should reference video content: robots.txt rule to opt out of AI training, asks question or observation. Must be short, one or two sentences, maybe fragment. No quotes, no markdown. Must avoid double hyphen. Ensure no em-dash, en-dash. Use straight quotes if needed. No URLs. So something like: "i tried adding the new robots.txt directive, but google still crawls my