I wanted to know if ChatGPT ever cites my blog. So I asked it a few questions my posts answer, like "Can I host WordPress on Vercel?". My post on that ranks on the first page of Google. ChatGPT cited Vercel's docs, WordPress.org and a couple of other blogs. Not mine.
Before trying to "optimize" anything, I checked something simpler: can ChatGPT read my site at all?
It couldn't. And robots.txt had nothing to do with it.
Short answer
Ask ChatGPT to open one of your pages and summarize it. If it says it got a 403, something is blocking it. That block can be in robots.txt, a security plugin, Cloudflare, or, in my case, a rule on the hosting server that I had no access to. You can confirm it in a minute with curl, and you can usually fix it without moving hosts.
The easiest test: ask ChatGPT to open a page
Open ChatGPT in a private window, logged out, so your own account doesn't change the answer. Then paste:
Open this exact page and summarize it in 3 bullet points. If you cannot open it, tell me exactly what error you got: https://yoursite.com/some-post/
Here's what I got for my ServerlessWP post:
Failed to fetch https://blog.yunuserturk.com/run-wordpress-on-vercel-serverlesswp/: (403) Forbidden
The end of a later 403 answer: ChatGPT couldn't open the post, so it explained the topic from other sources and said so.
That's the whole test. If ChatGPT can't fetch your page when you hand it the URL, it can't use that page as a source when someone else asks a question.
If you get a proper summary instead, your page is readable. Whether ChatGPT picks it as a source is a separate problem (more on that at the end).
Why robots.txt said everything was fine
The first thing everyone checks is robots.txt. Mine allowed every bot. No Disallow lines for GPTBot or anything else.
robots.txt is a polite request. It tells well-behaved crawlers what you'd like them to skip. It doesn't block anything. A real block happens on the server: the request comes in, something looks at it, and sends back a 403 before WordPress even loads.
So robots.txt can look perfect while the server is turning bots away.
Checking with curl
A crawler identifies itself with a user agent string. You can send the same string yourself and see what your server does with it. Compare a normal request with ChatGPT's search crawler:
curl -s -o /dev/null -w "%{http_code}\n" https://yoursite.com/robots.txt
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot/1.3" https://yoursite.com/robots.txt
The first should print 200. If the second prints 403, your server is blocking that crawler.
I ran it with a few user agents. Same page, same moment:
| User agent | Result |
|---|---|
| Normal browser, Googlebot, Bingbot | 200 |
| OAI-SearchBot (ChatGPT search) | 403 |
| ChatGPT-User (ChatGPT opening a link) | 403 |
| GPTBot (OpenAI training) | 403 |
| ClaudeBot, PerplexityBot | 403 |
| AhrefsBot, SemrushBot | 403 |
Google could read the site. ChatGPT, Claude and Perplexity couldn't. Neither could the two biggest SEO tools, which I hadn't noticed either.
Three OpenAI names are worth knowing. OAI-SearchBot builds the index ChatGPT search uses. ChatGPT-User is what fetches a page when a user asks about it. GPTBot collects training data. Blocking one doesn't block the others, and you can allow search while blocking training if you want.
Finding where the block was
The 403 page was the web server's generic error page, not a WordPress page. That already says WordPress isn't the one saying no. I went through the usual suspects:
- WordPress plugins: no security plugin with bot rules.
- .htaccess: only LiteSpeed Cache and the standard WordPress rewrite rules.
- ModSecurity (the firewall in the hosting panel): enabled, but its audit log was empty right after I triggered a 403. If ModSecurity had blocked the request, it would be logged there.
- Another site on the same server: same 403 for the same bots.
That last one settled it. A different site, owned by someone else, on the same server, behaved the same way. The rule was on the server itself, a list of "bad bots" the host maintains. On a shared hosting account you can't see or change it.
Testing more user agents showed how the list works: it matches on the name. GPTBot was blocked; a made-up "MyRandomBot" wasn't. Claude-User and Perplexity-User (the ones that fetch a page when a person asks) got through, while their crawlers didn't.
The fix
Option 1: ask your host. This is the obvious first step, and I did it. But shared hosts are often unwilling to change a server-wide rule for one customer, and you may wait days for an answer.
Option 2: put Cloudflare in front and pass the AI crawlers through. This is what I did. The blog now sits behind Cloudflare. A small Cloudflare Worker looks at each request: normal visitors pass through untouched; requests from the AI crawlers are forwarded to my server with a neutral user agent, so the host's list doesn't match them. Everything else about the site stays where it is.
The Worker, in the editor.
const AI_AGENTS = /oai-searchbot|chatgpt-user|gptbot|perplexitybot|claudebot/i;
export default {
async fetch(request) {
const ua = request.headers.get("user-agent") || "";
if (!AI_AGENTS.test(ua)) return fetch(request);
const headers = new Headers(request.headers);
headers.set("user-agent", "Mozilla/5.0 (compatible; Relay/1.0)");
return fetch(new Request(request, { headers }));
},
};
The Worker runs on a route for the blog's hostname only, with "fail open" on. Cloudflare's free plan covers 100,000 Worker requests a day, far more than a small blog needs, and with fail open, if that limit is ever reached, requests skip the Worker and go straight to the server instead of showing an error.
A few things that almost went wrong, so you can skip them:
-
Moving DNS to Cloudflare can break your email. When Cloudflare imports your records, it turns on its proxy (the orange cloud) for the root domain,
mail,ftpandwwwby default, and the import screen has "Proxy imported DNS records" ticked. Mail doesn't work through that proxy. I left only the blog proxied and everything else "DNS only". - Check your new Cloudflare settings for bot blocking. Bot Fight Mode would block the same crawlers again. I kept it off, and set the AI crawler settings to Allow.
- SSL mode: Full (strict), since the server already had a valid certificate.
If your site is already on Cloudflare and you still see 403s for AI crawlers, look at those bot settings first. That's the most common cause I've seen.
After the import, the root domain, ftp, mail and www were all set to Proxied.
The import screen ticks "Proxy imported DNS records" for you. Untick it.
AI crawler policies when adding the domain: all set to Allow.
Bot Fight Mode and AI Labyrinth off.
SSL/TLS set to Full (strict).
Did it work?
Same curl test, before and after, same moment:
| User agent | Direct to server | Through Cloudflare |
|---|---|---|
| OAI-SearchBot | 403 | 200 |
| ChatGPT-User | 403 | 200 |
| GPTBot | 403 | 200 |
| PerplexityBot | 403 | 200 |
| ClaudeBot | 403 | 200 |
| Googlebot | 200 | 200 |
And the test that matters: I asked ChatGPT to open the same post again. This time it gave me the title, author, date and a correct summary.
After the fix: ChatGPT opens the post and summarizes it.
One note if you test right after a DNS change: for a while some resolvers still point to the old server. My first ChatGPT retry still got a 403. A few minutes later it worked.
While I was in there: Bing
ChatGPT search leans on Bing's index, so I added the blog to Bing Webmaster Tools. Two surprises:
- My ServerlessWP post had been "Discovered but not crawled" since the day I published it.
- Two other posts were marked "Not indexed as this page is a redirect". The blog had a redirect problem weeks earlier, which I fixed, but Bing hadn't been back since.
Discovered on publish day, never crawled.
Bing still thought this post was a redirect.
I requested indexing for all three. Bing Webmaster also has an "AI Performance" report that shows how often Microsoft's AI products cite your pages. Mine showed zero for the last three months.
AI Performance: zero citations in three months.
If you've never opened Bing Webmaster Tools, it's worth ten minutes.
Checklist
- Ask ChatGPT, logged out, to open and summarize one of your pages. 403 or "couldn't access" means a block.
- Run curl with
OAI-SearchBot/1.3and compare with a normal request. - Check robots.txt, but don't stop there.
- Check security plugins and Cloudflare bot settings (Bot Fight Mode, AI crawler settings).
- Check whether another site on the same server behaves the same way. If it does, the rule is your host's.
- Ask your host, or route the AI crawlers through Cloudflare as above.
- Repeat step 1 to confirm.
- Add the site to Bing Webmaster Tools and check that your important pages are indexed.
Readable isn't the same as cited
Fixing this didn't make ChatGPT cite my posts. It made it possible. The same day, I asked nine questions my blog answers. ChatGPT now reads the pages, but for the topic questions it still cites other sources, and for "Can I run real WordPress on Vercel?" it still says "not natively", which my own test shows isn't true.
That's the second part of this experiment, and I'm measuring it every week. I'll write up what changes, with the numbers, in a follow-up post.
If ChatGPT can't read your site, or it describes your business wrong and you'd rather have someone sort it out, get in touch.
Originally published on my blog.
Top comments (0)