GPTBot in robots.txt: the hosting toggle developers need to check

Henrik Åberg — Mon, 07 Sep 2026 09:35:38 +0000

Your robots.txt may express an AI policy you did not write.

We checked the homepage and robots.txt of 9,037 live AI tools listed on directree on 6 and 7 September 2026. Of those, 945 explicitly disallow OpenAI’s GPTBot in its own user-agent group: 10.5% of the sample.

Treat AI crawler rules as deployment configuration. Review them when you change hosting, enable a CDN feature, adopt a starter template, or hand site operations to someone else.

Read the full research and methodology.

GPTBot, search, and user browsing are separate

A common configuration blocks model training while keeping a site available in AI-assisted search and browsing:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

These are separate crawlers with separate purposes. In our sample, 839 of the 945 sites that block GPTBot, or 88.8%, still allow OAI-SearchBot. That is a deliberate and useful distinction if your goal is to opt out of training while remaining eligible to be cited in ChatGPT search.

The same pattern appears across AI labs. ClaudeBot is explicitly blocked by 10.1% of the 9,037 tools, while Claude-SearchBot is blocked by just 0.1%. Google-Extended is blocked by 9.9%, but its purpose is also distinct from ordinary Google Search crawling.

Do not assume a broad-looking rule has the result you want. Check the actual crawler names and decide which capabilities you want to permit.

A safe way to review your file

Start by opening the public URL:

https://your-domain.example/robots.txt

Then look for three things:

A named crawler group, such as User-agent: GPTBot.
A Disallow: / directly inside that group.
A wildcard group, User-agent: *, that could affect all crawlers.

Our measurement only counts a site as blocking GPTBot when the named GPTBot group itself contains Disallow: /. This matters because ordinary technical exclusions are widespread. Only 31 sites in the 9,037-site sample, or 0.3%, block every crawler outright. Meanwhile, 44% have a path-level Disallow rule in a wildcard group, often for an admin area or API route. That is normal site hygiene, not necessarily an AI policy.

Keep sensitive paths protected through authentication and application controls too. robots.txt is a crawler instruction, not access control.

Why hosting provider settings deserve attention

Hosting correlated strongly with whether a tool blocked GPTBot. Of 3,749 Cloudflare-hosted sites, 22.4% explicitly blocked it. On Vercel, 5.2% of 2,556 sites did. Netlify was 4.0% across 273 sites, Fastly 3.9% across 361, and CloudFront 3.4% across 328.

That makes Cloudflare-hosted tools more than four times as likely to block GPTBot as Vercel-hosted tools in this sample.

The likely explanation is operational rather than philosophical. A hosting control that offers to block AI bots can write or manage crawler policy at scale. That is convenient, but it can make a consequential content policy feel like a harmless performance or security toggle.

Whenever you enable one of these settings, verify the deployed result rather than relying on the dashboard label. Fetch robots.txt after deployment and include it in your release checklist.

Make the choice explicit

A small policy file is easier to maintain than an inherited one. If you want to block training but preserve discoverability, state that clearly with named groups. If you want to allow everything, avoid leaving old template rules behind. If you want a complete block, understand that wildcard rules can affect conventional search crawlers too.

The broader data shows this is becoming more common: 13.6% of 2,900 domains registered in 2026 block GPTBot, compared with 6.9% of 145 domains registered in 2023. That makes it even more important to know whether your policy is intentional.

For the complete crawler breakdown, hosting results, and methodology, see How many AI tools block GPTBot.

DEV Community: Henrik Åberg

GPTBot in robots.txt: the hosting toggle developers need to check

GPTBot, search, and user browsing are separate

A safe way to review your file

Why hosting provider settings deserve attention

Make the choice explicit