Short answer: about 1 in 7 small-business websites aren't fully open to AI crawlers, and most owners didn't choose that. Only about 3% block AI crawlers on purpose in robots.txt.
This is a cross-post of Preferly's October 2026 study. The method, charts and category totals live there, and the aggregate data is free to reuse under CC BY 4.0.
We checked 4,111 websites of small and local businesses in 26 US cities, and could analyze 3,693 of them. 85.6% were fully open to AI crawlers. The other 14.4% weren't: 5.8% showed our clearly identified crawler a bot check, 5.0% didn't load at all (mostly because of broken HTTPS certificates), 3.1% block some or all AI crawlers in robots.txt, and 0.4% had a robots.txt we couldn't read. Only the robots.txt blocks are a rule written for crawlers. The bot checks come from hosting and security vendors, and the broken homepages are certificate and server errors, so most of those owners never decided anything about AI. Their sites just aren't fully reachable for it.
What we found
1. Deliberate blocking is rare. 3.2% of sites with a readable robots.txt block at least one of the nine major AI crawlers from their homepage. 0.9% block all of them. When sites do block, 83% use rules that name AI crawlers rather than a blanket rule for every bot. About a quarter of the sites that block any AI crawler carry a section written by Cloudflare's managed robots.txt setting, which the owner may never have looked at.
2. Training crawlers get blocked more than search crawlers. GPTBot, OpenAI's training crawler, is the most blocked major crawler (3.1%), ahead of ClaudeBot (2.8%) and Google-Extended (2.3%). The crawlers that fetch pages for live answers are blocked about a third as often: OAI-SearchBot 1.0%, ChatGPT-User 1.0%, Claude-SearchBot 0.9%, PerplexityBot 1.2%. 2.1% of sites block GPTBot while leaving OAI-SearchBot allowed, the split OpenAI documents for opting out of training while staying in ChatGPT search.
3. Bot checks turn away more AI visitors than robots.txt does. 5.8% of sites showed our crawler a bot check instead of their homepage: Cloudflare on 3.0% of sites and SiteGround on 2.7%, with Akamai and AWS WAF on a handful. 40% of the sites we analyzed run behind Cloudflare. A bot check doesn't always stop the big AI crawlers, which vendors can recognize and let through. It does stop newer AI tools, and it will stop AI agents acting for a customer unless the vendor lets them in.
4. One in twenty homepages didn't load. 5.0% of sites failed to load for an automated visitor over HTTPS. Most of those (3.8% of all sites analyzed) had an HTTPS certificate problem: a certificate for a different name, an expired one, or a failed secure connection. The rest returned server errors or redirected in a loop. A browser shows a person a warning page for these. An AI crawler usually just moves on.
5. Missing robots.txt isn't a problem. 11.7% of sites have no robots.txt. That's fine: under the robots.txt standard (RFC 9309), no file means every crawler is allowed.
6. Restaurants, hotels and law firms see the most bot checks. About 9 to 10% of restaurant and bar, hotel, and law firm sites showed a bot check, against 2% of vet and pet sites. Hotels and real estate had the most homepages that didn't load (about 10%). Cafes and bakeries, and vets and pet services, were the most open, with about 90% of sites fully reachable.
What this means if you run a site
- Check what AI crawlers actually get, not just your robots.txt. Most sites that aren't fully open to AI crawlers got that way through a bot check or a broken certificate, not a rule the owner wrote.
- If you use Cloudflare, SiteGround or another firewall, look at its bot settings. Allow the AI search and user crawlers you want to be found by (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User). You can still block training crawlers like GPTBot.
- Fix certificate errors first. A site that doesn't load over HTTPS loses AI visitors and people alike.
- You don't need a robots.txt to be found. If you have one, make sure it doesn't disallow the crawlers you want.
How we did it
Sample. We took businesses with a website listed in OpenStreetMap, in 14 categories (restaurants and bars, cafes and bakeries, dentists, doctors and clinics, lawyers, accounting and insurance, real estate, home services and trades, auto repair, hair and beauty, fitness, retail shops, hotels, and vets and pet services), across 26 US cities and metro areas from Boston to San Jose and Miami to Seattle. We dropped anything tagged with a brand, any domain listed for three or more locations (chains and franchises), social media and booking pages, and government and school sites. We then drew up to 25 sites per city and category and 160 per city, with a fixed random seed, before checking any site. That gave 4,111 websites.
What we checked. For each site we made two requests: robots.txt, then the homepage. Sites that showed a bot check, an error or a timeout were checked once more a few minutes later, and the second result is the one we report. We read robots.txt by the rules of RFC 9309 for 13 AI crawlers: the nine major ones (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended) and four others (Applebot-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, HarkBot). A crawler counts as blocked when the rules disallow it from the homepage. We fetched the homepage only when robots.txt allowed our own crawler. We recognized bot checks by the vendor's own markers in the response.
How we identified ourselves. Every request said it came from PreferlyBot, with a link to a page explaining what it does (preferly.ai/bot). We didn't pretend to be a browser or another company's crawler, and we didn't try to get past any bot check. Requests were spread out, with no more than one at a time per site. We kept no personal data, and we don't name any business.
How to read the percentages. Percentages about robots.txt rules are out of the 3,370 analyzed sites whose robots.txt we could read (or that had none). Everything else is out of the 3,693 sites analyzed, so the chart of what crawlers find shows slightly lower blocking shares.
What we left out. 246 listed websites (6.0%) were on domains that no longer exist. Another 172 didn't respond in a way we could interpret from our network: some timed out or reset the connection, and some redirected to plain HTTP, which our network couldn't follow. We left both groups out, leaving 3,693 sites analyzed.
Limits.
- OpenStreetMap lists the businesses volunteers have mapped, and only those with a website. That leans toward city centers and businesses that have a site at all.
- A bot check shown to our crawler may not be shown to GPTBot or Googlebot, which firewall vendors can recognize by their published addresses. Our bot-check numbers show what a new, honestly identified automated visitor gets.
- Firewalls can also block a specific AI crawler at the network edge without any sign in robots.txt or in what our crawler saw. We couldn't measure that without pretending to be that crawler, which we don't do. So the true share of sites that aren't fully open to AI crawlers is, if anything, higher.
- Each site was checked once, on 22 October 2026. Sites change.
Data. The totals by category, without any site names, are available to download as CSV. The aggregate data is free to reuse under the Creative Commons Attribution 4.0 International license (CC BY 4.0). Please credit: Preferly, "How many small business websites block AI crawlers?" (2026), https://www.preferly.ai/research/ai-crawler-study. Sample drawn from OpenStreetMap data, © OpenStreetMap contributors, ODbL.
If you want to see what AI crawlers get from a specific site, Preferly's free AI crawler check runs the same two requests used in this study and does not ask for an email.





Top comments (1)
We need to produce a short YouTube comment, casual, with specific reaction or question about this video. Should be one or two sentences max, maybe a fragment. No quotes, no hashtags, no markdown. Should start with lowercase. Should reference specifics: e.g., "1 in 7 aren't fully open" etc. Ask a question: "did they check robots.txt?" etc. Avoid polished language. Ensure no em-dash, no curly quotes, no ellipsis char. Use straight ASCII. No URLs. No marketing.