Most founders assume that if a page loads in a browser and shows up in Google, an AI assistant can read it too. We scanned 383 products from Show HN with a script we keep in the repo. 27% were unreadable to crawlers, and 17% were blocked from Claude outright.
27% of 383 launches could not be read by a crawler
These are shipped products. Landing pages, docs, pricing, the whole thing. They render fine for a human with a Chrome tab. Fetch them the way a crawler does and better than a quarter of them return something a machine cannot use.
The failure is not exotic. It is client-side rendering with no server response worth parsing, or an interstitial, or a bot filter that decides an unfamiliar user agent is an attack. The founder never sees it because the founder always arrives with a real browser and a real cookie jar.
17% blocked Claude, and almost none of it was a decision
Here is the part that surprised me. Of those 383 products, 17% were blocked from Claude. Not throttled, not rate limited on a bad day. Blocked.
If you ask the founders, most of them will tell you they never made that call. There is no board meeting where a two-person team decides to exclude an AI assistant from reading their marketing site. The rule came from somewhere else.
robots.txt says yes, the edge says no
This is the actual mechanism, and it is worth checking on your own domain today.
Your robots.txt is permissive. You wrote it, or your framework wrote it, and it allows everything. Then a CDN bot management rule, a WAF preset, or a "block AI scrapers" toggle someone flipped during a scare month intercepts the request before your app ever hears about it. The two layers disagree, and the layer that wins is not the one in your repo.
So the file you can read in your editor is not the file that governs behavior. That is why reading robots.txt is not a test. Sending a request with the crawler's user agent is a test.
| What you check | What it actually tells you |
|---|---|
| Page loads in your browser | A logged-in human with JS can see it |
| robots.txt allows the bot | Your repo intends to allow the bot |
| Google Search Console coverage | Google specifically got through |
| Fetch with the crawler's user agent | Whether that crawler got through |
The fourth row is the only one that answers the question, and it is the row almost nobody runs.
Analytics cannot show you traffic that never arrived
This is a distribution bug that lives in your infrastructure and is invisible in your dashboards. There is no line in Plausible for "assistant tried to read us and got a 403." The request failed upstream, so the visit never existed, so there is nothing to count and nothing to alert on.
Compare that to a normal outage. If your checkout breaks you get support tickets within an hour. If your site is unreadable to an AI crawler you get silence, indefinitely, and you interpret the silence as low demand.
Our own sitemap left out the three pages we most wanted indexed
I would like to report that we found this problem in other people's repos only. We did not.
Our sitemap omitted three of our highest-intent pages, including the free check page and the proof wall. Both were indexable routes with no noindex, no auth, nothing unusual. They were simply absent from the file whose entire job is telling crawlers what exists. Generated sitemap, route filter, silent omission.
That is a one-line class of bug. It is also the kind of thing that stays broken for months because finding it requires someone to diff the sitemap against the route table, and that task never wins a sprint.
Ranking and being citable are different failure modes
SEO aims at a position in a results page. AI visibility aims at being readable and citable by an assistant. They overlap, but they fail independently.
A page can rank perfectly well on Google and still be blocked from an AI crawler by a bot filter, because Googlebot has been on the allowlist since before the filter existed and the newer agents have not. Your rank tracker will show green the entire time. That is the trap: the instrument you trust is measuring a different pipe.
A 60 item audit list is not a fix
The standard remedy here is a crawler that produces a report. You get a PDF, the PDF goes into Notion, and six weeks later the title tags are unchanged. I have done this. Most people reading this have done this.
The reason is not laziness. It is that a recommendation creates work and a diff removes it. Meta descriptions, title tags, structured data, robots and crawler access, sitemap entries: all of these are edits to files that already live in your repo. A pull request against them takes a minute to review. A list of the same edits takes an afternoon to implement and therefore never gets implemented.
We build a tool that opens those as PRs (Viewfy), which is how the sitemap bug above got caught, but you do not need a tool to run the scan. Curl with a user agent string and a loop over your routes will find most of this.
What this scan cannot tell you
Several honest limits.
It is a single snapshot. A bot filter that returned 403 during our scan window might pass on retry, and rate limiting can look identical to a block if you only ask once.
It does not prove intent. We say most of the Claude blocks were not deliberate because founders say so and because the robots.txt files contradict the edge behavior, not because we surveyed all 383 teams.
It does not measure lost revenue. Being readable is a precondition for being cited, not a cause of it. A crawlable page with nothing worth quoting stays uncited.
And Show HN is a skewed sample: early launches, heavy on JS frameworks, heavy on free CDN tiers with aggressive defaults. A cohort of five year old SaaS companies would likely fail differently, probably less on rendering and more on stale structured data.
What the numbers do support is narrow and checkable: for 383 real shipped products, 27% could not be read by a crawler and 17% were closed to Claude, and you can re-run that test against your own domain in about ten minutes.
What does your site return when you fetch it with an AI crawler's user agent, and did you write that rule or did your CDN?
Full method and the scan script live on our blog at viewfy.ai.
Top comments (0)