Most founders assume the rules in their robots.txt are rules they chose. Across 156 Show HN launches from August 2026, 23 sites (14.7%) disallow Anthropic's ClaudeBot while leaving OpenAI's OAI-SearchBot allowed, and 0 sites (0.0%) do the reverse. In a cohort of 156 independent founders shipping independent products, a coin-flip preference does not come out 23 to zero.
23 to 0 is not a preference, it is a paste
The perfect one-sidedness is the tell. Something is writing the same decision into everyone's repo.
That something is Cloudflare's managed robots.txt. Of the 24 sites in the cohort that disallow any AI crawler, 23 (95.8%) serve that managed file, identifiable by its Content-Signal preamble. The file declares search=yes in the same breath that it disallows ClaudeBot, GPTBot and Google-Extended. The site is telling AI search engines it wants to be found and telling three named crawlers to go away.
The written blocks are uniform in a way hand-editing never produces. ClaudeBot, GPTBot and Google-Extended are each disallowed on the same 24 sites (15.4%). OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot and PerplexityBot are each disallowed on 1 site (0.6%). Exactly one site in 156 runs a blanket disallow, and it is the only block in the whole dataset that reads like a deliberate choice.
Cloudflare wrote the block, not the founder
Cloudflare authored the blocks on 33 of the 156 sites (21.2%). That is 50.8% of the 59 Cloudflare-fronted launches, and Cloudflare fronts 37.8% of the cohort overall.
The cost is concrete. atlasmotion.com launched motors for drones and robotics with 406 words of readable homepage HTML and one point on Hacker News. That page is unreachable to ChatGPT, Claude, Google AI Overviews and Perplexity at once. zipbox.ai shipped 2,141 words about Firecracker VMs for agents and has the same four-engine verdict.
One founder in r/SEO put it plainly: "Then I ran a curl on /robots.txt and saw this block that I definitely didn't write".
Half of the blocks are theater
On 12 of the 23 managed-file sites, the origin still hands ClaudeBot a 200. The block exists only on paper, which is worse than either alternative: crawlers that respect robots.txt stay away from a page the server was happy to serve. You get the traffic loss of a block with none of the protection, and nothing in your logs looks broken.
Twelve rows you can read line by line
Rows where robots.txt names GPTBot, ClaudeBot and Google-Extended but the 403 column is empty are paper-only blocks. Rows where the 403 column is long and the robots column is empty are silent CDN blocks. Rows with both are invisible to all four engines.
| host | server | CF managed robots.txt | robots.txt disallows | HTTP 403 to | engines unreachable | words in raw HTML |
|---|---|---|---|---|---|---|
| atlasmotion.com | cloudflare | yes | GPTBot, ClaudeBot, Google-Extended | OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot | ChatGPT, Claude, Google AI Overviews, Perplexity | 406 |
| zipbox.ai | cloudflare | yes | GPTBot, ClaudeBot, Google-Extended | OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot | ChatGPT, Claude, Google AI Overviews, Perplexity | 2141 |
| portfoliovideo.com | cloudflare | yes | GPTBot, ClaudeBot, Google-Extended | OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot | ChatGPT, Claude, Google AI Overviews, Perplexity | 2554 |
| fosbury.ai | cloudflare | yes | GPTBot, ClaudeBot, Google-Extended | none | Claude, Google AI Overviews | 46 |
| cochat.ai | cloudflare | yes | GPTBot, ClaudeBot, Google-Extended | none | Claude, Google AI Overviews | 1571 |
| imagetovideoai.tools | cloudflare | yes | GPTBot, ClaudeBot, Google-Extended | none | Claude, Google AI Overviews | 1215 |
| scalequest.io | cloudflare | yes | GPTBot, ClaudeBot, Google-Extended | none | Claude, Google AI Overviews | 5 |
| usecharming.com | cloudflare | yes | none | none | Claude, Google AI Overviews, Perplexity | 1241 |
| spacescience.tech | cloudflare | yes | none | OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot | ChatGPT, Claude, Perplexity | 5 |
| today.spqrk.net | Apache | no | none | ClaudeBot | Claude | 1163 |
| reelang.com | cloudflare | yes | GPTBot, ClaudeBot, Google-Extended | OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended | Claude, Google AI Overviews | 3 |
| modelplane.dev | cloudflare | yes | GPTBot, ClaudeBot, Google-Extended | OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot | ChatGPT, Claude, Google AI Overviews, Perplexity | 393 |
reelang.com is the row to stare at: three readable words in the un-executed HTML, a managed robots.txt disallowing three crawlers, and a server refusing seven user-agents including Google-Extended. A language-learning product with speech-analyzed native speaker video, and an assistant asked about it has nothing to read.
The ChatGPT block is silent; the Google block is written down
The two layers block opposite engines. robots.txt in this cohort forbids ClaudeBot, GPTBot and Google-Extended. The CDN and WAF layer refuses OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot and PerplexityBot at the network layer, where nothing is declared.
15 sites (9.6%) returned a non-200 to at least one AI crawler. OAI-SearchBot and PerplexityBot were refused on 14 each (9.0%). Google-Extended was refused on only 2 (1.3%), and on 13 of those 15 sites (86.7%) Google-Extended got a 200 while every other AI crawler got a 403.
Stack the layers and honesty goes lopsided:
- Google AI Overviews: unreachable on 25 sites (16.0%), only 4.0% of those blocks silent.
- ChatGPT: unreachable on 15 sites (9.6%), 93.3% silent.
- Perplexity: unreachable on 15 sites (9.6%), 93.3% silent.
- Claude: worst off at 28 sites (17.9%), at least 14.3% silent, the rest declared.
A commenter in r/SEO described the intent behind the split: "Cloudflare blocks the training bots, not the ones that handle search/appearance on the platform. For example, Cloudflare will block GPTBot and allow OAI-SearchBot". The measurement says the bot-fight rules underneath do not honor that split.
Your host decides your visibility more than your craft does
Cloudflare-fronted launches are invisible at 47.5% of 59 sites. Vercel-fronted launches are invisible at 2.6% of 39. That is a ratio of 18.5, and Cloudflare accounts for 82.4% of every invisible site while making up 37.8% of the cohort. nginx sat at 10.0% invisible across 20 sites.
Each stack fails differently. Cloudflare's failure is the block: 8.5% JavaScript-only, only 6.8% missing robots.txt. Vercel's failure is absence: 28.2% serve no robots.txt at all, nginx 35.0%. The 10 launches on shared app-platform subdomains were 20.0% invisible, 20.0% JavaScript-only shells, and 80.0% with no robots.txt.
Cohort-wide: 21.8% of the 156 are invisible to at least one engine and 71.8% are fully clean. 15.4% blocked in robots.txt, 9.6% refused at the network layer, 5.8% ship an empty JavaScript shell, 19.2% serve no robots.txt, 9.0% are thin HTML. Median readable words in raw HTML: 687. Sitemap declared: 67.3%.
Four checks you can run before your coffee cools
-
curl https://yoursite.com/robots.txtand look for a Content-Signal preamble. 21.2% of the cohort has one; 50.8% of Cloudflare-fronted sites do. If it disallows ClaudeBot, GPTBot and Google-Extended while declaringsearch=yes, you did not write that. -
curl -A "OAI-SearchBot" -I https://yoursite.com, then repeat for ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot and Google-Extended. 9.6% of the cohort returns a non-200 to at least one, and 93.3% of the ChatGPT blocks appear nowhere in robots.txt. -
curl -s https://yoursite.com | wc -won the un-executed HTML. 5.8% of launches ship a shell that is empty until JavaScript runs. Median was 687 words. scalequest.io shipped five, reelang.com three. - Confirm robots.txt and sitemap.xml exist at all. 19.2% serve no robots.txt, only 67.3% declare a sitemap. On Vercel that miss rate was 28.2%, on nginx 35.0%.
What this data cannot say
Homepage only, one request per user-agent, one two-day slice of Show HN, so a rate limit or transient 403 can read as a permanent block. We made no attempt to verify that a user-agent string belonged to the vendor it claims. We did not test Cloudflare defaults, only outcomes.
This study also counts ClaudeBot among retrieval crawlers and GPTBot as training-only, and blocking ClaudeBot does not remove a site from Claude entirely: the 23 sites disallowing ClaudeBot still allow Claude-User and Claude-SearchBot, so exposure is reduced, not zero. Claude still came out worst overall because the network layer refused Claude-User and Claude-SearchBot on 14 sites each, separately from anything robots.txt said.
Method, briefly: Hacker News Algolia search_by_date with tags=show_hn, newest first, no upvote filter, deduped by host, code hosts and app stores and socials and doc hosts dropped. 274 posts scanned down to 170 hosts, 166 reachable, 156 on their own domain and 10 on shared app-platform subdomains. Nine requests per homepage, no crawl beyond it. Spot checks reproduced by hand with curl -A on atlasmotion.com and zipbox.ai.
Run check 1 on your own domain and tell me what came back: did you write that file, or did your CDN?
Full dataset, per-host rows and the recompute script live on the Viewfy blog.
Correction, added after publication
A founder in the cohort pushed back on the network layer numbers, and he was right. Our probes sent crawler user agents from an ordinary IP, so a 403 in that setup measures how a CDN treats an impersonator, not the real crawler. Verified bots calling from their published IP ranges typically pass. The robots.txt findings in this post are unaffected, that file is public text anyone can read. But treat every "unreachable" and WAF number above as "unmeasurable from the outside", not "blocked". We have changed the methodology so outbound claims only use the robots.txt layer.
Top comments (1)
Worth adding why this one is so easy to miss: the Content-Signal preamble you're fingerprinting only exists in the response coming off the edge, never in the repo, so grepping your codebase for GPTBot returns nothing and any CI lint on the committed file passes clean. It can also flip after launch with no deploy on your side, so a single curl on ship day isn't a durable answer. We ended up running that fetch on a schedule in Viewfy for that reason.