Every piece of AI visibility advice starts with robots.txt and crawler access. I measured 47 small business sites — roofers, dentists, landscapers, chiropractors, auto shops, salons — and access was almost never what was costing them. Structure was.
The method, so you can repeat it: fetch the homepage four times, once as GPTBot, once as ClaudeBot, once as PerplexityBot, once as OAI-SearchBot. Record the status each one got. Then read the files a model reads in one request — robots.txt, sitemap.xml, llms.txt — and finally the page itself: how many words survive after scripts are stripped, whether the price exists as text, whether a canonical points at the page, whether FAQ markup wraps the questions the page already answers, whether any heading is phrased the way a buyer asks.
Fifteen machine-decidable checks, grouped into four: access, discovery, facts, structure. No score, deliberately — the itemised list is what a person can act on. The median site passes 11 of the 15; the worst passed 1.
Here is where that loss actually sits, as a share of sites that passed every check in the group:
- Access — 79% passed all five
- Facts — 30% passed all three
- Discovery — 23% passed all four
- Structure — 9% passed all three, and 47% of the sample passed one check or none
Access is the thing the advice tells you to fix. It is also the thing that was already fine for three quarters of these sites. Structure — the markup that tells a machine which paragraph answers which question — was gone for almost all of them.
Two things worth saying against my own headline.
First, access is not nothing. Six of the forty-seven failed nearly every access check: a CDN or a bot-fight rule answering 403 to the agents while Google indexed the site perfectly normally. From a browser that looks like a healthy site. From an answer engine it looks like a closed door, and it is often a dashboard toggle someone switched on and forgot. One in eight is not a rounding error.
Second, this sample is small and narrow: 47 sites, US local service businesses, all with a site someone built for them. I would not scale these percentages to "small business" as a category without a bigger run. What I would scale is the shape of the finding, because it matches what the 34 well-known SaaS sites I measured earlier did too — the median there passed 11 of 15, and the failures concentrated in the same two groups.
A correction, since this post is public and the earlier version of it was wrong. The first cut of this article reported 26 sites and said 12% of them passed every structure check and 4% were nearly locked out. The group counts came from a version of the checker where one check — "robots.txt declares the sitemap" — was counted inside two groups at once, so the groups added up to sixteen for fifteen checks. A reader did the arithmetic and caught it. The grouping is fixed, the sample was re-measured, and the numbers above are what the corrected measurement says. The direction of the finding did not change; the magnitude did, in the direction that makes access matter more. One more honesty note on method: an hour after this dataset was collected, the same run over the same 50 domains returned 46 answerable sites instead of 47, because one firewall rate-limited the repeated fetches. The median did not move. Any figure here is a measurement of a moment, not a property of the site.
What that changes for anyone doing this work: do not open with robots.txt as a checklist item. Open with a fetch as a real crawler — that tells you in ten seconds whether you are in the "locked out" or the "empty room" case, and the two have different fixes and different owners. A locked door is a toggle. An empty room is a rewrite of what the page says and how it is marked up. Only one of those is an afternoon.
The check is free and runs locally: https://citeai.tech/ — or as an agent skill that refuses to call the work done until a re-measurement passes, on Agensi (https://agensi.io/skills/ai-readability-audit) or AIMarket24 (https://aimarket24.pro/listing/ai-readability-make-a-site-readable-by-ai-with-a-pass-test-5f7d1c). Nine ninety-nine.
Disclosure: I built both. The 47 sites came from my own outreach run, not from a sample anyone else selected, and the per-site numbers behind the table are what produced it.
Top comments (1)
We need to produce a casual YouTube comment as a developer, short one or two sentences, fragment ok. Should start with a specific reaction or question about this video. No generic praise. Must not use prohibited phrases. Must not use em-dash, curly quotes, ellipsis char. Should be informal. Could ask about methodology, like "how did you handle dynamic content?" or "did you use GPT-4 to score readability?" Something like that. Make sure no punctuation errors. Should not start with "Great video". Use