If a client asks "why doesn't ChatGPT ever mention us?", the first thing to check is whether ChatGPT can read their site at all. It's a two-minute check, and it's often the problem.
Three OpenAI crawlers, three different jobs
From OpenAI's crawler documentation:
| User agent | What it does | If you block it |
|---|---|---|
OAI-SearchBot |
Finds sites for ChatGPT search | You won't be shown in ChatGPT search answers |
GPTBot |
Collects content that may train OpenAI models | Your content is opted out of training |
ChatGPT-User |
Visits a page when a user asks ChatGPT to | robots.txt may not apply, since a user triggered it |
They're independent. Blocking GPTBot does not remove you from ChatGPT search, and a lot of sites that "blocked AI" in a panic blocked the wrong one.
The rules
Appear in ChatGPT search, opt out of training:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
Or allow everything, which is what a default WordPress + Yoast setup does:
User-agent: *
Disallow:
An empty Disallow: means nothing is blocked. OpenAI says robots.txt changes take about 24 hours to apply to search.
Where to edit it in WordPress:
- Yoast: SEO → Tools → File editor
- Rank Math: General Settings → Edit robots.txt
Add custom rules below the plugin's generated block.
The check most people skip: the firewall
robots.txt can say "allow" while something else blocks the bot before it ever reads the file:
- Cloudflare has settings to block AI bots. Check your bot and AI crawler settings in the dashboard.
- WordPress security plugins often block unknown or "bad" bots by user agent.
- Hosting firewalls (ModSecurity rules) sometimes do the same.
A quick test for user-agent based blocking:
curl -I -A "OAI-SearchBot" https://example.com/
curl -I -A "GPTBot" https://example.com/
You want 200 OK. A 403 or a challenge page means something is blocking by user agent. This won't catch IP-based bot verification (Cloudflare checks real crawler IPs), so look at the dashboard too.
Don't forget Bing
Microsoft Copilot answers come from Bing's index. Two quick wins:
- Add the site to Bing Webmaster Tools (import from Google Search Console) and submit the sitemap.
- Enable IndexNow. Rank Math has it built in; Microsoft has an IndexNow plugin for WordPress.
Bing Webmaster Tools also has an AI Performance report showing how often your pages are cited in Copilot.
And Google?
Google says there are no extra requirements for AI Overviews or AI Mode: no special schema, no AI text files. If the page is indexed and eligible for a snippet, it's eligible.
Crawlers are step one
Access gets you considered. Getting recommended depends on clear content, consistent business details and third-party mentions. I wrote the full guide for Pakistani businesses here:
👉 How to Get Your Business Recommended by ChatGPT and Google AI
Written by Mursil Hamayoun, WordPress developer and founder of Pixelscape, Lahore.
Top comments (0)