DEV Community

Cover image for robots.txt for AI Crawlers: OAI-SearchBot vs GPTBot (and the Firewall Check Most People Skip)
M.Saqib
M.Saqib

Posted on

robots.txt for AI Crawlers: OAI-SearchBot vs GPTBot (and the Firewall Check Most People Skip)

If a client asks "why doesn't ChatGPT ever mention us?", the first thing to check is whether ChatGPT can read their site at all. It's a two-minute check, and it's often the problem.

Three OpenAI crawlers, three different jobs

From OpenAI's crawler documentation:

User agent What it does If you block it
OAI-SearchBot Finds sites for ChatGPT search You won't be shown in ChatGPT search answers
GPTBot Collects content that may train OpenAI models Your content is opted out of training
ChatGPT-User Visits a page when a user asks ChatGPT to robots.txt may not apply, since a user triggered it

They're independent. Blocking GPTBot does not remove you from ChatGPT search, and a lot of sites that "blocked AI" in a panic blocked the wrong one.

The rules

Appear in ChatGPT search, opt out of training:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /
Enter fullscreen mode Exit fullscreen mode

Or allow everything, which is what a default WordPress + Yoast setup does:

User-agent: *
Disallow:
Enter fullscreen mode Exit fullscreen mode

An empty Disallow: means nothing is blocked. OpenAI says robots.txt changes take about 24 hours to apply to search.

Where to edit it in WordPress:

  • Yoast: SEO → Tools → File editor
  • Rank Math: General Settings → Edit robots.txt

Add custom rules below the plugin's generated block.

The check most people skip: the firewall

robots.txt can say "allow" while something else blocks the bot before it ever reads the file:

  • Cloudflare has settings to block AI bots. Check your bot and AI crawler settings in the dashboard.
  • WordPress security plugins often block unknown or "bad" bots by user agent.
  • Hosting firewalls (ModSecurity rules) sometimes do the same.

A quick test for user-agent based blocking:

curl -I -A "OAI-SearchBot" https://example.com/
curl -I -A "GPTBot" https://example.com/
Enter fullscreen mode Exit fullscreen mode

You want 200 OK. A 403 or a challenge page means something is blocking by user agent. This won't catch IP-based bot verification (Cloudflare checks real crawler IPs), so look at the dashboard too.

Don't forget Bing

Microsoft Copilot answers come from Bing's index. Two quick wins:

  1. Add the site to Bing Webmaster Tools (import from Google Search Console) and submit the sitemap.
  2. Enable IndexNow. Rank Math has it built in; Microsoft has an IndexNow plugin for WordPress.

Bing Webmaster Tools also has an AI Performance report showing how often your pages are cited in Copilot.

And Google?

Google says there are no extra requirements for AI Overviews or AI Mode: no special schema, no AI text files. If the page is indexed and eligible for a snippet, it's eligible.

Crawlers are step one

Access gets you considered. Getting recommended depends on clear content, consistent business details and third-party mentions. I wrote the full guide for Pakistani businesses here:

👉 How to Get Your Business Recommended by ChatGPT and Google AI

Written by Mursil Hamayoun, WordPress developer and founder of Pixelscape, Lahore.

Top comments (0)