Almost every guide about getting cited by AI opens the same way.
Your robots.txt is blocking GPTBot. Your CDN is returning 403 to Perplexity. Fix that before you do anything else.
I wanted to know whether that was actually true. So I ran the test against eleven sites that already rank well and already show up in AI answers: Ahrefs, Moz, Semrush, Search Engine Journal, Screaming Frog, Stripe, Shopify, GitHub, Cloudflare, Dev.to, and Vercel.
I expected to find a few of them blocking. Maybe the big ones, protecting content behind a wall.
Not one of them was. Zero out of eleven. And the third finding was the one that actually surprised me.
The four crawlers people keep confusing
Before the results, one clarification, because most of the bad advice online comes from people mixing these up. They are not the same thing and they do very different jobs.
| User agent | Who runs it | What it does |
|---|---|---|
GPTBot |
OpenAI | Trains models. Being blocked here does not stop ChatGPT from citing you. |
OAI-SearchBot |
OpenAI | Indexes pages for ChatGPT search. This is the one that decides citations. |
ChatGPT-User |
OpenAI | Fetches a single page when a user pastes a URL or asks a live question. |
PerplexityBot |
Perplexity | Indexes and cites. |
ClaudeBot |
Anthropic | Crawls for Claude. |
Google-Extended |
Controls Gemini training and grounding. Does not control AI Overviews. | |
CCBot |
Common Crawl | Public web archive used by many downstream models. |
The one that matters most for getting cited is OAI-SearchBot, and it is the one almost nobody mentions. Everyone writes about GPTBot, which mostly affects training, not citations.
How the test works
Two checks per site. Both take under a minute.
Check one — what does robots.txt say?
curl -s https://example.com/robots.txt | grep -iE "gptbot|oai-searchbot|perplexitybot|claudebot|google-extended|ccbot"
If you get output with Disallow: / under a bot's name, that bot is blocked.
Check two — what happens when the crawler actually shows up?
This one matters more, because robots.txt is a polite request. A CDN, a WAF, or a bot-management toggle can ignore it completely and return 403 to anything it does not like. Many managed hosts do exactly this without telling you.
for ua in "GPTBot/1.2" "OAI-SearchBot/1.0" "PerplexityBot/1.0" "ClaudeBot/1.0"; do
echo -n "$ua -> "
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; $ua; +https://example.com)" \
https://example.com
done
A 200 means the crawler gets in. A 403 means it gets the door slammed in its face, no matter what robots.txt promised.
Finding 1: nothing is blocked. Nothing.
Eleven out of eleven sites allow GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended, and CCBot. All of them, all six, no exceptions. No wildcard Disallow: / anywhere.
Then I sent each AI user agent to each home page. Every single combination came back HTTP 200.
| Domain | robots.txt | GPTBot | OAI-SearchBot | PerplexityBot | ClaudeBot |
|---|---|---|---|---|---|
| ahrefs.com | allow all | 200 | 200 | 200 | 200 |
| moz.com | allow all | 200 | 200 | 200 | 200 |
| semrush.com | allow all | 200 | 200 | 200 | 200 |
| searchenginejournal.com | allow all | 200 | 200 | 200 | 200 |
| screamingfrog.co.uk | allow all | 200 | 200 | 200 | 200 |
| stripe.com | allow all | 200 | 200 | 200 | 200 |
| shopify.com | allow all | 200 | 200 | 200 | 200 |
| github.com | allow all | 200 | 200 | 200 | 200 |
| cloudflare.com | allow all | 200 | 200 | 200 | 200 |
| dev.to | allow all | 200 | 200 | 200 | 200 |
| vercel.com | allow all | 200 | 200 | 200 | 200 |
Fifty-five requests. Fifty-five 200s. Not a single 403, not a single challenge page, not a single rate limit.
I want to be careful about what this proves. These are established sites with engineering teams who have looked at this question. It does not mean your host is fine. Custom CDN rules, security plugins, and "block bad bots" toggles are exactly where this breaks, and they break silently. That is why the ninety-second check is still worth running on your own domain.
But it does mean the blanket warning is wrong. The default state of the modern web is permissive.
Finding 2: eight of eleven publish an llms.txt
llms.txt is a plain text file at yourdomain.com/llms.txt. It is not an official standard. No crawler is required to read it. It is just a short Markdown summary of what your site is, what it covers, and where the important pages live.
Eight of the eleven serve one:
| Domain | llms.txt | Size |
|---|---|---|
| Cloudflare | ✅ | 17 KB (plus a 169 KB full version) |
| Stripe | ✅ | 69.8 KB |
| GitHub | ✅ | 28.6 KB |
| Semrush | ✅ | 14.9 KB |
| Shopify | ✅ | 12.5 KB (plus 18.8 KB full version) |
| Vercel | ✅ | 5.4 KB |
| Dev.to | ✅ | 2.8 KB |
| Ahrefs | ❌ 404 | — |
| Screaming Frog | ❌ 404 | — |
| Moz | ❌ 403 on that path | — |
Here is what they actually say, in their own words:
# Stripe
> Stripe is a technology company that provides financial infrastructure
> for businesses to accept payments, grow revenue, and access capital.
# GitHub
> GitHub is a developer platform for building, shipping, and maintaining
> software.
# Cloudflare
> Cloudflare is a global cloud connectivity company operating in 335+ cities.
# DEV Community
> A Forem community where people share knowledge and participate in
> discussions.
# Semrush
> Semrush is a brand visibility management platform that helps marketers
> audit and improve their online presence.
Every single one follows the same shape. A # heading with the company name. A > blockquote with one or two sentences describing what the company does. Then a list of links.
That is the whole format. No schema, no JavaScript, no build step, no plugin.
Look at what those sentences have in common. Technology company that provides financial infrastructure. Developer platform for building software. Global cloud connectivity company. Brand visibility management platform.
Not one of them says "we empower businesses to unlock their potential." They say what the company is, in a sentence a stranger can repeat back.
Finding 3: the uncomfortable one
If access is not the bottleneck — and it clearly is not, at least among sites that are already succeeding — then something else decides who gets cited.
Look at the eight that publish an llms.txt. It is not that they are big. Ahrefs is big and does not have one. It is not that they are technical. It is that all eight wrote down, in one place, in plain sentences, what they are and what they do.
The newest ones treat it as a product decision, not an SEO trick. Stripe's file is 69 kilobytes of clear statements about what Stripe is, who it is for, and where each piece of documentation lives. That is a business writing down its own facts in public.
Which brings up the part most guides skip.
You can allow every crawler in the world and still get ignored. The crawler arrives, reads your page, and finds a sentence that says "we help businesses unlock their potential." That is not a fact a retrieval system can extract. It is a slogan, and slogans are noise.
Access is a checkbox you can fix once. Clarity is the work, and it never ends.
Crawled is not the same as indexed
One more distinction this test does not cover, and it trips up a lot of people.
Every site above returned 200. That proves the crawler got in the front door. It does not prove the page ended up in an index. Getting crawled and getting indexed are two separate events, and the second one can fail for reasons that have nothing to do with access.
Search Console has a status for exactly this: Crawled – currently not indexed. It means Google fetched the page and then decided not to store it. Nothing was blocked. No 403, no robots.txt rule, no permission problem. The page simply was not considered worth indexing.
That is a different problem, with a different fix, and it comes up constantly on Blogger, where the same article is reachable at ?m=1, with UTM parameters, and through feed URLs, and canonical selection gets messy.
I wrote a separate walkthrough for that specific situation, because it is not something crawler access testing can answer:
Crawled – Currently Not Indexed on Blogger: What to Check (2026) →
The order matters. Check access first, using the commands in this post. Then check canonical selection. Then look at the page itself. Fixing indexing before you have fixed access is wasted effort, because you cannot index what a crawler cannot fetch.
What this test does not prove
Three honest limitations, because the data is only worth as much as its caveats.
The sample is small and biased. Eleven sites, all of them already established, all of them English-language. A startup site on a cheap shared host with an unknown security plugin may behave completely differently.
Home pages are not content pages. I tested the root domain. Individual article pages can have different rules, and article pages are the ones that get cited.
llms.txt has no measured effect here. I did not test whether the eight sites with llms.txt get cited more often. I did not test whether the three without get cited less. That correlation would need far more data. What I can say is that eight of eleven chose to publish one, and all of them are free to do.
What I would check, in this order
1. Actually run the two commands above. Do not assume. Most managed hosts allow everything by default, but custom CDN rules and bot-management toggles quietly break this. Ninety seconds, and it either clears the question or finds a real problem.
2. Write an llms.txt. Upload it to your root. One heading, one blockquote, a short list of your most important pages with plain descriptions. Twenty minutes.
Here is a template you can copy, fill in, and upload:
# Your Company
> One sentence. What you are, who it is for, what you sell.
> Write it so a stranger could repeat it back.
## Main pages
- [Home](https://example.com): what the product does
- [Pricing](https://example.com/pricing): plans and costs
- [Docs](https://example.com/docs): how to use it
## About
- [Company](https://example.com/about): who we are
- [Contact](https://example.com/contact): how to reach us
Replace every placeholder with a real sentence about your own site. Save as llms.txt, upload to your domain root, confirm it loads at yourdomain.com/llms.txt.
3. Read your own home page as if you were a stranger who knows nothing about you. Can you find, in plain sentences: what this is, who it is for, what it costs, and where to go next? If those four facts are buried under adjectives, no crawler permission will save you.
4. Test a content page, not just your home page. Pick your best article or product page and run the same two commands on it. That is where citations actually come from.
I turned all of this into a scored checklist with the exact commands and a 100-point breakdown. It is free and there is no email gate before the download:
AI Search Readiness Checklist →
One last thing
I would genuinely like to know if anyone finds a site that does block. Eleven out of eleven is a small sample, and I would rather my conclusion be wrong in public than quietly wrong in private.
Drop the domain in the comments and I will add it to the table. If someone finds twenty sites that block, the whole premise of this post changes, and I would want to know.
Top comments (0)