DEV Community: Mike Viewfy

We scanned 383 Show HN products. 27% were unreadable to crawlers, 17% blocked Claude (and nobody wrote that rule)

Mike Viewfy — Mon, 10 Aug 2026 00:59:40 +0000

Most founders assume that if a page loads in a browser and shows up in Google, an AI assistant can read it too. We scanned 383 products from Show HN with a script we keep in the repo. 27% were unreadable to crawlers, and 17% were blocked from Claude outright.

27% of 383 launches could not be read by a crawler

These are shipped products. Landing pages, docs, pricing, the whole thing. They render fine for a human with a Chrome tab. Fetch them the way a crawler does and better than a quarter of them return something a machine cannot use.

The failure is not exotic. It is client-side rendering with no server response worth parsing, or an interstitial, or a bot filter that decides an unfamiliar user agent is an attack. The founder never sees it because the founder always arrives with a real browser and a real cookie jar.

17% blocked Claude, and almost none of it was a decision

Here is the part that surprised me. Of those 383 products, 17% were blocked from Claude. Not throttled, not rate limited on a bad day. Blocked.

If you ask the founders, most of them will tell you they never made that call. There is no board meeting where a two-person team decides to exclude an AI assistant from reading their marketing site. The rule came from somewhere else.

robots.txt says yes, the edge says no

This is the actual mechanism, and it is worth checking on your own domain today.

Your robots.txt is permissive. You wrote it, or your framework wrote it, and it allows everything. Then a CDN bot management rule, a WAF preset, or a "block AI scrapers" toggle someone flipped during a scare month intercepts the request before your app ever hears about it. The two layers disagree, and the layer that wins is not the one in your repo.

So the file you can read in your editor is not the file that governs behavior. That is why reading robots.txt is not a test. Sending a request with the crawler's user agent is a test.

What you check	What it actually tells you
Page loads in your browser	A logged-in human with JS can see it
robots.txt allows the bot	Your repo intends to allow the bot
Google Search Console coverage	Google specifically got through
Fetch with the crawler's user agent	Whether that crawler got through

The fourth row is the only one that answers the question, and it is the row almost nobody runs.

Analytics cannot show you traffic that never arrived

This is a distribution bug that lives in your infrastructure and is invisible in your dashboards. There is no line in Plausible for "assistant tried to read us and got a 403." The request failed upstream, so the visit never existed, so there is nothing to count and nothing to alert on.

Compare that to a normal outage. If your checkout breaks you get support tickets within an hour. If your site is unreadable to an AI crawler you get silence, indefinitely, and you interpret the silence as low demand.

Our own sitemap left out the three pages we most wanted indexed

I would like to report that we found this problem in other people's repos only. We did not.

Our sitemap omitted three of our highest-intent pages, including the free check page and the proof wall. Both were indexable routes with no noindex, no auth, nothing unusual. They were simply absent from the file whose entire job is telling crawlers what exists. Generated sitemap, route filter, silent omission.

That is a one-line class of bug. It is also the kind of thing that stays broken for months because finding it requires someone to diff the sitemap against the route table, and that task never wins a sprint.

Ranking and being citable are different failure modes

SEO aims at a position in a results page. AI visibility aims at being readable and citable by an assistant. They overlap, but they fail independently.

A page can rank perfectly well on Google and still be blocked from an AI crawler by a bot filter, because Googlebot has been on the allowlist since before the filter existed and the newer agents have not. Your rank tracker will show green the entire time. That is the trap: the instrument you trust is measuring a different pipe.

A 60 item audit list is not a fix

The standard remedy here is a crawler that produces a report. You get a PDF, the PDF goes into Notion, and six weeks later the title tags are unchanged. I have done this. Most people reading this have done this.

The reason is not laziness. It is that a recommendation creates work and a diff removes it. Meta descriptions, title tags, structured data, robots and crawler access, sitemap entries: all of these are edits to files that already live in your repo. A pull request against them takes a minute to review. A list of the same edits takes an afternoon to implement and therefore never gets implemented.

We build a tool that opens those as PRs (Viewfy), which is how the sitemap bug above got caught, but you do not need a tool to run the scan. Curl with a user agent string and a loop over your routes will find most of this.

What this scan cannot tell you

Several honest limits.

It is a single snapshot. A bot filter that returned 403 during our scan window might pass on retry, and rate limiting can look identical to a block if you only ask once.

It does not prove intent. We say most of the Claude blocks were not deliberate because founders say so and because the robots.txt files contradict the edge behavior, not because we surveyed all 383 teams.

It does not measure lost revenue. Being readable is a precondition for being cited, not a cause of it. A crawlable page with nothing worth quoting stays uncited.

And Show HN is a skewed sample: early launches, heavy on JS frameworks, heavy on free CDN tiers with aggressive defaults. A cohort of five year old SaaS companies would likely fail differently, probably less on rendering and more on stale structured data.

What the numbers do support is narrow and checkable: for 383 real shipped products, 27% could not be read by a crawler and 17% were closed to Claude, and you can re-run that test against your own domain in about ten minutes.

What does your site return when you fetch it with an AI crawler's user agent, and did you write that rule or did your CDN?

Full method and the scan script live on our blog at viewfy.ai.

I probed 156 Show HN launches: 23 block Claude, 0 block OpenAI (almost none wrote that file)

Mike Viewfy — Fri, 07 Aug 2026 22:30:52 +0000

Most founders assume the rules in their robots.txt are rules they chose. Across 156 Show HN launches from August 2026, 23 sites (14.7%) disallow Anthropic's ClaudeBot while leaving OpenAI's OAI-SearchBot allowed, and 0 sites (0.0%) do the reverse. In a cohort of 156 independent founders shipping independent products, a coin-flip preference does not come out 23 to zero.

23 to 0 is not a preference, it is a paste

The perfect one-sidedness is the tell. Something is writing the same decision into everyone's repo.

That something is Cloudflare's managed robots.txt. Of the 24 sites in the cohort that disallow any AI crawler, 23 (95.8%) serve that managed file, identifiable by its Content-Signal preamble. The file declares search=yes in the same breath that it disallows ClaudeBot, GPTBot and Google-Extended. The site is telling AI search engines it wants to be found and telling three named crawlers to go away.

The written blocks are uniform in a way hand-editing never produces. ClaudeBot, GPTBot and Google-Extended are each disallowed on the same 24 sites (15.4%). OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot and PerplexityBot are each disallowed on 1 site (0.6%). Exactly one site in 156 runs a blanket disallow, and it is the only block in the whole dataset that reads like a deliberate choice.

Cloudflare wrote the block, not the founder

Cloudflare authored the blocks on 33 of the 156 sites (21.2%). That is 50.8% of the 59 Cloudflare-fronted launches, and Cloudflare fronts 37.8% of the cohort overall.

The cost is concrete. atlasmotion.com launched motors for drones and robotics with 406 words of readable homepage HTML and one point on Hacker News. That page is unreachable to ChatGPT, Claude, Google AI Overviews and Perplexity at once. zipbox.ai shipped 2,141 words about Firecracker VMs for agents and has the same four-engine verdict.

One founder in r/SEO put it plainly: "Then I ran a curl on /robots.txt and saw this block that I definitely didn't write".

Half of the blocks are theater

On 12 of the 23 managed-file sites, the origin still hands ClaudeBot a 200. The block exists only on paper, which is worse than either alternative: crawlers that respect robots.txt stay away from a page the server was happy to serve. You get the traffic loss of a block with none of the protection, and nothing in your logs looks broken.

Twelve rows you can read line by line

Rows where robots.txt names GPTBot, ClaudeBot and Google-Extended but the 403 column is empty are paper-only blocks. Rows where the 403 column is long and the robots column is empty are silent CDN blocks. Rows with both are invisible to all four engines.

host	server	CF managed robots.txt	robots.txt disallows	HTTP 403 to	engines unreachable	words in raw HTML
atlasmotion.com	cloudflare	yes	GPTBot, ClaudeBot, Google-Extended	OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot	ChatGPT, Claude, Google AI Overviews, Perplexity	406
zipbox.ai	cloudflare	yes	GPTBot, ClaudeBot, Google-Extended	OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot	ChatGPT, Claude, Google AI Overviews, Perplexity	2141
portfoliovideo.com	cloudflare	yes	GPTBot, ClaudeBot, Google-Extended	OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot	ChatGPT, Claude, Google AI Overviews, Perplexity	2554
fosbury.ai	cloudflare	yes	GPTBot, ClaudeBot, Google-Extended	none	Claude, Google AI Overviews	46
cochat.ai	cloudflare	yes	GPTBot, ClaudeBot, Google-Extended	none	Claude, Google AI Overviews	1571
imagetovideoai.tools	cloudflare	yes	GPTBot, ClaudeBot, Google-Extended	none	Claude, Google AI Overviews	1215
scalequest.io	cloudflare	yes	GPTBot, ClaudeBot, Google-Extended	none	Claude, Google AI Overviews	5
usecharming.com	cloudflare	yes	none	none	Claude, Google AI Overviews, Perplexity	1241
spacescience.tech	cloudflare	yes	none	OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot	ChatGPT, Claude, Perplexity	5
today.spqrk.net	Apache	no	none	ClaudeBot	Claude	1163
reelang.com	cloudflare	yes	GPTBot, ClaudeBot, Google-Extended	OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended	Claude, Google AI Overviews	3
modelplane.dev	cloudflare	yes	GPTBot, ClaudeBot, Google-Extended	OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot	ChatGPT, Claude, Google AI Overviews, Perplexity	393

reelang.com is the row to stare at: three readable words in the un-executed HTML, a managed robots.txt disallowing three crawlers, and a server refusing seven user-agents including Google-Extended. A language-learning product with speech-analyzed native speaker video, and an assistant asked about it has nothing to read.

The ChatGPT block is silent; the Google block is written down

The two layers block opposite engines. robots.txt in this cohort forbids ClaudeBot, GPTBot and Google-Extended. The CDN and WAF layer refuses OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot and PerplexityBot at the network layer, where nothing is declared.

15 sites (9.6%) returned a non-200 to at least one AI crawler. OAI-SearchBot and PerplexityBot were refused on 14 each (9.0%). Google-Extended was refused on only 2 (1.3%), and on 13 of those 15 sites (86.7%) Google-Extended got a 200 while every other AI crawler got a 403.

Stack the layers and honesty goes lopsided:

Google AI Overviews: unreachable on 25 sites (16.0%), only 4.0% of those blocks silent.
ChatGPT: unreachable on 15 sites (9.6%), 93.3% silent.
Perplexity: unreachable on 15 sites (9.6%), 93.3% silent.
Claude: worst off at 28 sites (17.9%), at least 14.3% silent, the rest declared.

A commenter in r/SEO described the intent behind the split: "Cloudflare blocks the training bots, not the ones that handle search/appearance on the platform. For example, Cloudflare will block GPTBot and allow OAI-SearchBot". The measurement says the bot-fight rules underneath do not honor that split.

Your host decides your visibility more than your craft does

Cloudflare-fronted launches are invisible at 47.5% of 59 sites. Vercel-fronted launches are invisible at 2.6% of 39. That is a ratio of 18.5, and Cloudflare accounts for 82.4% of every invisible site while making up 37.8% of the cohort. nginx sat at 10.0% invisible across 20 sites.

Each stack fails differently. Cloudflare's failure is the block: 8.5% JavaScript-only, only 6.8% missing robots.txt. Vercel's failure is absence: 28.2% serve no robots.txt at all, nginx 35.0%. The 10 launches on shared app-platform subdomains were 20.0% invisible, 20.0% JavaScript-only shells, and 80.0% with no robots.txt.

Cohort-wide: 21.8% of the 156 are invisible to at least one engine and 71.8% are fully clean. 15.4% blocked in robots.txt, 9.6% refused at the network layer, 5.8% ship an empty JavaScript shell, 19.2% serve no robots.txt, 9.0% are thin HTML. Median readable words in raw HTML: 687. Sitemap declared: 67.3%.

Four checks you can run before your coffee cools

curl https://yoursite.com/robots.txt and look for a Content-Signal preamble. 21.2% of the cohort has one; 50.8% of Cloudflare-fronted sites do. If it disallows ClaudeBot, GPTBot and Google-Extended while declaring search=yes, you did not write that.
curl -A "OAI-SearchBot" -I https://yoursite.com, then repeat for ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot and Google-Extended. 9.6% of the cohort returns a non-200 to at least one, and 93.3% of the ChatGPT blocks appear nowhere in robots.txt.
curl -s https://yoursite.com | wc -w on the un-executed HTML. 5.8% of launches ship a shell that is empty until JavaScript runs. Median was 687 words. scalequest.io shipped five, reelang.com three.
Confirm robots.txt and sitemap.xml exist at all. 19.2% serve no robots.txt, only 67.3% declare a sitemap. On Vercel that miss rate was 28.2%, on nginx 35.0%.

What this data cannot say

Homepage only, one request per user-agent, one two-day slice of Show HN, so a rate limit or transient 403 can read as a permanent block. We made no attempt to verify that a user-agent string belonged to the vendor it claims. We did not test Cloudflare defaults, only outcomes.

This study also counts ClaudeBot among retrieval crawlers and GPTBot as training-only, and blocking ClaudeBot does not remove a site from Claude entirely: the 23 sites disallowing ClaudeBot still allow Claude-User and Claude-SearchBot, so exposure is reduced, not zero. Claude still came out worst overall because the network layer refused Claude-User and Claude-SearchBot on 14 sites each, separately from anything robots.txt said.

Method, briefly: Hacker News Algolia search_by_date with tags=show_hn, newest first, no upvote filter, deduped by host, code hosts and app stores and socials and doc hosts dropped. 274 posts scanned down to 170 hosts, 166 reachable, 156 on their own domain and 10 on shared app-platform subdomains. Nine requests per homepage, no crawl beyond it. Spot checks reproduced by hand with curl -A on atlasmotion.com and zipbox.ai.

Run check 1 on your own domain and tell me what came back: did you write that file, or did your CDN?
Full dataset, per-host rows and the recompute script live on the Viewfy blog.

Correction, added after publication

A founder in the cohort pushed back on the network layer numbers, and he was right. Our probes sent crawler user agents from an ordinary IP, so a 403 in that setup measures how a CDN treats an impersonator, not the real crawler. Verified bots calling from their published IP ranges typically pass. The robots.txt findings in this post are unaffected, that file is public text anyone can read. But treat every "unreachable" and WAF number above as "unmeasurable from the outside", not "blocked". We have changed the methodology so outbound claims only use the robots.txt layer.