<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Preferly Team</title>
    <description>The latest articles on DEV Community by Preferly Team (@preferly).</description>
    <link>https://dev.to/preferly</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4203079%2Fbc6f050b-f7bc-4341-8ab6-0065ffe70cc5.png</url>
      <title>DEV Community: Preferly Team</title>
      <link>https://dev.to/preferly</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/preferly"/>
    <language>en</language>
    <item>
      <title>Do these three things before you ever start paying an AEO agency</title>
      <dc:creator>Preferly Team</dc:creator>
      <pubDate>Thu, 19 Nov 2026 09:10:35 +0000</pubDate>
      <link>https://dev.to/preferly/do-these-three-things-before-you-ever-start-paying-an-aeo-agency-p7o</link>
      <guid>https://dev.to/preferly/do-these-three-things-before-you-ever-start-paying-an-aeo-agency-p7o</guid>
      <description>&lt;p&gt;This is a cross-post of &lt;a href="https://www.preferly.ai/blog/before-hiring-an-aeo-agency" rel="noopener noreferrer"&gt;Preferly's guide&lt;/a&gt; on what to do before hiring an AEO agency.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Short answer:&lt;/strong&gt; before you pay an AEO agency, get a complete audit of your current AI visibility, do the quick fixes yourself (this usually takes a few hours), then wait 2-4 weeks and run the same audit again. That second baseline is what you take to the agency, so you can measure what they actually change.&lt;/p&gt;

&lt;p&gt;A founder recently shared that their AEO agency was a complete fail. The milder version is common too: an agency bills for a month of work, visibility ticks up, and some of that jump would have happened anyway once the site could be read and the obvious errors were cleaned up. Skip the three steps below and you pay an agency to do things you could have done in an afternoon, and they take credit for the low-hanging fruit.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Get a complete audit of your current AI visibility
&lt;/h2&gt;

&lt;p&gt;A useful audit is a snapshot you can repeat, not a slide deck. It should tell you, in writing, where you stand today on each of these.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The assistants your customers use.&lt;/strong&gt; ChatGPT, Gemini, Google AI Overviews and AI Mode, Perplexity and Claude don't always name the same businesses. Check the ones your customers actually open.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The questions a customer would ask.&lt;/strong&gt; Don't only search your own name. Ask category and problem questions without naming yourself ("best family dentist in South Tampa", "emergency plumber near Westchase open Sunday"), then ask about you by name. Repeat each question a few times in a private window, logged out. Answers vary from run to run, so one missing mention isn't a baseline.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Crawler access.&lt;/strong&gt; If AI tools can't load your pages, they can't quote them. Check &lt;code&gt;robots.txt&lt;/code&gt; for rules that block search crawlers such as OAI-SearchBot, Claude-SearchBot or PerplexityBot. Check whether a firewall or bot check turns identified crawlers away, which is common on Cloudflare and some hosts. Confirm the homepage loads over HTTPS with a valid certificate.&lt;/p&gt;

&lt;p&gt;This is more common than it feels. In Preferly's &lt;a href="https://www.preferly.ai/research/ai-crawler-study" rel="noopener noreferrer"&gt;study of 3,693 small and local business websites&lt;/a&gt; in 26 US cities, 14.4% (about 1 in 7) weren't fully open to AI crawlers, and most owners didn't choose that: 5.8% showed a bot check, 5.0% didn't load (mostly HTTPS failures), and 3.1% blocked some or all AI crawlers in &lt;code&gt;robots.txt&lt;/code&gt;. Preferly's &lt;a href="https://www.preferly.ai/ai-crawler-check" rel="noopener noreferrer"&gt;free AI crawler check&lt;/a&gt; runs the same two requests used in that study.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Listings and reviews.&lt;/strong&gt; AI answers often lean on Google Business Profile, maps, review sites and directories. If those pages show old hours, the wrong category, a closed location, or a different name from your website, the audit should record it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Schema and on-site content.&lt;/strong&gt; Do your key facts (name, address, hours, services, area served) appear in the HTML, not only after JavaScript runs? Is there LocalBusiness structured data, and does it match the visible page?&lt;/p&gt;

&lt;p&gt;You can gather a first pass with tools like Preferly's &lt;a href="https://www.preferly.ai/start" rel="noopener noreferrer"&gt;free check&lt;/a&gt; or others, or by hand in a few private chats. Write down the questions, the models, who was named, which URLs were cited, and any crawler or listing problems, so you can run the same audit again later.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Do all the quick fixes based on your audit
&lt;/h2&gt;

&lt;p&gt;Most of what turns up is not a content strategy problem. It's a blocked crawler, a certificate warning, a stale listing, or a homepage that never states what you do in plain text. This shouldn't take more than a few hours.&lt;/p&gt;

&lt;p&gt;Typical fixes, in the order they usually pay off:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Let the search crawlers in.&lt;/strong&gt; If &lt;code&gt;robots.txt&lt;/code&gt; disallows OAI-SearchBot or the other search crawlers you care about, allow them. You can still block training crawlers such as GPTBot. This is often a few minutes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Relax the bot check for verified AI crawlers.&lt;/strong&gt; If Cloudflare or your host shows a challenge page to identified crawlers, adjust the bot settings so search crawlers can load the page, and leave protection in place for everyone else. This is usually a short visit to the dashboard.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fix the HTTPS certificate&lt;/strong&gt; if the homepage doesn't load. An AI crawler that hits a certificate error usually just moves on. Renewing or replacing a certificate is often quick; if the domain or DNS is wrong, it may take longer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Make your listings match your website.&lt;/strong&gt; Hours, address, phone, name, category, and whether you're open. Start with Google Business Profile, then the directories the first audit cited. This can fill an afternoon if several listings are out of date.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Put the facts in the HTML.&lt;/strong&gt; Name, location, hours, services, and who you serve should be readable without JavaScript. Add or correct LocalBusiness schema so it matches what's on the page.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Correct the pages AI already cites&lt;/strong&gt; when they contradict your site, including third-party pages you can claim or update.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Save a copy of what you changed and when. You'll need that list when you read the second audit.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Wait 2-4 weeks, then do a second audit
&lt;/h2&gt;

&lt;p&gt;Fixes don't show up in AI answers the same day. Crawlers have to revisit the pages, and some answers still come from older copies or from training data that only changes when a new model ships.&lt;/p&gt;

&lt;p&gt;OpenAI says a &lt;code&gt;robots.txt&lt;/code&gt; change can reach its search systems in about 24 hours, but being crawled and then cited can take longer. Listing and third-party updates often take longer than a site change. That's why we say wait 2-4 weeks: long enough for a re-crawl and for answers to refresh.&lt;/p&gt;

&lt;p&gt;Then run the same audit again: same assistants, same questions, same method. Compare whether you're named more often on the customer questions, whether answers about you match your website, which sources are cited now, and whether crawlers can load the pages they couldn't before. That second snapshot is your new baseline. Everything after it is what an agency can fairly be judged on.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to use the second baseline in an agency contract
&lt;/h2&gt;

&lt;p&gt;Bring the two audits (before the fixes, and 2-4 weeks after) to the first serious conversation. Ask the agency to treat the second one as day zero.&lt;/p&gt;

&lt;p&gt;Measure the same questions, on the same assistants, on a fixed schedule. Agree in writing what "improvement" means: how often you're named on a set of customer questions, whether the facts about you stay accurate, and which sources those answers cite. Ask them to keep the question list stable, and to show the raw answers, not just a score.&lt;/p&gt;

&lt;p&gt;Be wary of a pitch that starts work without a baseline, or that wants credit for the easy crawler and listing fixes you already made. Those were yours.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to ask an AEO agency before signing
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Will you use my second baseline as the starting point, and keep the same questions and assistants?&lt;/li&gt;
&lt;li&gt;What, exactly, will you measure, and how often will I see the raw answers?&lt;/li&gt;
&lt;li&gt;Which assistants are in scope, and which are not?&lt;/li&gt;
&lt;li&gt;What work is included in the first 30 days, and what is out of scope (ads, review generation, and so on)?&lt;/li&gt;
&lt;li&gt;How do you treat the quick technical fixes (&lt;code&gt;robots.txt&lt;/code&gt;, bot checks, certificates, listings)? If those are still open, who does them, and are they billed as strategy?&lt;/li&gt;
&lt;li&gt;What will you need from me, and what can you not do if a third-party site refuses to update?&lt;/li&gt;
&lt;li&gt;How do you report when answers vary from run to run?&lt;/li&gt;
&lt;li&gt;What happens if visibility doesn't move after the agreed period?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the answers are vague, wait. A good agency should be comfortable being measured against a baseline you already have.&lt;/p&gt;

&lt;p&gt;You don't need an agency to find out where you stand. Tools like Preferly's &lt;a href="https://www.preferly.ai/start" rel="noopener noreferrer"&gt;free check&lt;/a&gt; or others will get you the first audit. The important part is that you do it, fix what's obvious, and measure again before you pay someone else to take the credit.&lt;/p&gt;

&lt;p&gt;The full version, with internal guides for robots.txt, Cloudflare, HTTPS and ChatGPT citations, is on &lt;a href="https://www.preferly.ai/blog/before-hiring-an-aeo-agency" rel="noopener noreferrer"&gt;preferly.ai&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>seo</category>
      <category>marketing</category>
      <category>webdev</category>
    </item>
    <item>
      <title>How many small business websites block AI crawlers?</title>
      <dc:creator>Preferly Team</dc:creator>
      <pubDate>Thu, 19 Nov 2026 07:05:08 +0000</pubDate>
      <link>https://dev.to/preferly/how-many-small-business-websites-block-ai-crawlers-51m6</link>
      <guid>https://dev.to/preferly/how-many-small-business-websites-block-ai-crawlers-51m6</guid>
      <description>&lt;p&gt;&lt;strong&gt;Short answer:&lt;/strong&gt; about 1 in 7 small-business websites aren't fully open to AI crawlers, and most owners didn't choose that. Only about 3% block AI crawlers on purpose in robots.txt.&lt;/p&gt;

&lt;p&gt;This is a cross-post of &lt;a href="https://www.preferly.ai/research/ai-crawler-study" rel="noopener noreferrer"&gt;Preferly's October 2026 study&lt;/a&gt;. The method, charts and category totals live there, and the aggregate data is free to reuse under &lt;a href="https://creativecommons.org/licenses/by/4.0/" rel="noopener noreferrer"&gt;CC BY 4.0&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;We checked 4,111 websites of small and local businesses in 26 US cities, and could analyze 3,693 of them. 85.6% were fully open to AI crawlers. The other 14.4% weren't: 5.8% showed our clearly identified crawler a bot check, 5.0% didn't load at all (mostly because of broken HTTPS certificates), 3.1% block some or all AI crawlers in robots.txt, and 0.4% had a robots.txt we couldn't read. Only the robots.txt blocks are a rule written for crawlers. The bot checks come from hosting and security vendors, and the broken homepages are certificate and server errors, so most of those owners never decided anything about AI. Their sites just aren't fully reachable for it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we found
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4cnzvmhyldnvckdt905x.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4cnzvmhyldnvckdt905x.png" alt="What an AI crawler finds on a small business website" width="800" height="427"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Deliberate blocking is rare.&lt;/strong&gt; 3.2% of sites with a readable robots.txt block at least one of the nine major AI crawlers from their homepage. 0.9% block all of them. When sites do block, 83% use rules that name AI crawlers rather than a blanket rule for every bot. About a quarter of the sites that block any AI crawler carry a section written by Cloudflare's managed robots.txt setting, which the owner may never have looked at.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Training crawlers get blocked more than search crawlers.&lt;/strong&gt; GPTBot, OpenAI's training crawler, is the most blocked major crawler (3.1%), ahead of ClaudeBot (2.8%) and Google-Extended (2.3%). The crawlers that fetch pages for live answers are blocked about a third as often: OAI-SearchBot 1.0%, ChatGPT-User 1.0%, Claude-SearchBot 0.9%, PerplexityBot 1.2%. 2.1% of sites block GPTBot while leaving OAI-SearchBot allowed, the split OpenAI documents for opting out of training while staying in ChatGPT search.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwo7426qp4mzjrd4e8iuu.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwo7426qp4mzjrd4e8iuu.png" alt="Which AI crawlers small business sites block in robots.txt" width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Bot checks turn away more AI visitors than robots.txt does.&lt;/strong&gt; 5.8% of sites showed our crawler a bot check instead of their homepage: Cloudflare on 3.0% of sites and SiteGround on 2.7%, with Akamai and AWS WAF on a handful. 40% of the sites we analyzed run behind Cloudflare. A bot check doesn't always stop the big AI crawlers, which vendors can recognize and let through. It does stop newer AI tools, and it will stop AI agents acting for a customer unless the vendor lets them in.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftz2f21ceg2fvfrsesw6e.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftz2f21ceg2fvfrsesw6e.png" alt="Bot checks by vendor" width="800" height="400"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. One in twenty homepages didn't load.&lt;/strong&gt; 5.0% of sites failed to load for an automated visitor over HTTPS. Most of those (3.8% of all sites analyzed) had an HTTPS certificate problem: a certificate for a different name, an expired one, or a failed secure connection. The rest returned server errors or redirected in a loop. A browser shows a person a warning page for these. An AI crawler usually just moves on.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Missing robots.txt isn't a problem.&lt;/strong&gt; 11.7% of sites have no robots.txt. That's fine: under the robots.txt standard (&lt;a href="https://www.rfc-editor.org/rfc/rfc9309.html" rel="noopener noreferrer"&gt;RFC 9309&lt;/a&gt;), no file means every crawler is allowed.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmuguk4u4fzx7i5ahgezo.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmuguk4u4fzx7i5ahgezo.png" alt="Does the site have a robots.txt?" width="800" height="360"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. Restaurants, hotels and law firms see the most bot checks.&lt;/strong&gt; About 9 to 10% of restaurant and bar, hotel, and law firm sites showed a bot check, against 2% of vet and pet sites. Hotels and real estate had the most homepages that didn't load (about 10%). Cafes and bakeries, and vets and pet services, were the most open, with about 90% of sites fully reachable.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9s78b7s7aibrjcckzax1.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9s78b7s7aibrjcckzax1.png" alt="By category" width="800" height="560"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What this means if you run a site
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Check what AI crawlers actually get, not just your robots.txt.&lt;/strong&gt; Most sites that aren't fully open to AI crawlers got that way through a bot check or a broken certificate, not a rule the owner wrote.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;If you use Cloudflare, SiteGround or another firewall, look at its bot settings.&lt;/strong&gt; Allow the AI search and user crawlers you want to be found by (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User). You can still block training crawlers like GPTBot.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fix certificate errors first.&lt;/strong&gt; A site that doesn't load over HTTPS loses AI visitors and people alike.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You don't need a robots.txt to be found.&lt;/strong&gt; If you have one, make sure it doesn't disallow the crawlers you want.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How we did it
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Sample.&lt;/strong&gt; We took businesses with a website listed in OpenStreetMap, in 14 categories (restaurants and bars, cafes and bakeries, dentists, doctors and clinics, lawyers, accounting and insurance, real estate, home services and trades, auto repair, hair and beauty, fitness, retail shops, hotels, and vets and pet services), across 26 US cities and metro areas from Boston to San Jose and Miami to Seattle. We dropped anything tagged with a brand, any domain listed for three or more locations (chains and franchises), social media and booking pages, and government and school sites. We then drew up to 25 sites per city and category and 160 per city, with a fixed random seed, before checking any site. That gave 4,111 websites.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What we checked.&lt;/strong&gt; For each site we made two requests: robots.txt, then the homepage. Sites that showed a bot check, an error or a timeout were checked once more a few minutes later, and the second result is the one we report. We read robots.txt by the rules of RFC 9309 for 13 AI crawlers: the nine major ones (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended) and four others (Applebot-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, HarkBot). A crawler counts as blocked when the rules disallow it from the homepage. We fetched the homepage only when robots.txt allowed our own crawler. We recognized bot checks by the vendor's own markers in the response.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How we identified ourselves.&lt;/strong&gt; Every request said it came from PreferlyBot, with a link to a page explaining what it does (&lt;a href="https://www.preferly.ai/bot" rel="noopener noreferrer"&gt;preferly.ai/bot&lt;/a&gt;). We didn't pretend to be a browser or another company's crawler, and we didn't try to get past any bot check. Requests were spread out, with no more than one at a time per site. We kept no personal data, and we don't name any business.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How to read the percentages.&lt;/strong&gt; Percentages about robots.txt rules are out of the 3,370 analyzed sites whose robots.txt we could read (or that had none). Everything else is out of the 3,693 sites analyzed, so the chart of what crawlers find shows slightly lower blocking shares.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What we left out.&lt;/strong&gt; 246 listed websites (6.0%) were on domains that no longer exist. Another 172 didn't respond in a way we could interpret from our network: some timed out or reset the connection, and some redirected to plain HTTP, which our network couldn't follow. We left both groups out, leaving 3,693 sites analyzed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Limits.&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;OpenStreetMap lists the businesses volunteers have mapped, and only those with a website. That leans toward city centers and businesses that have a site at all.&lt;/li&gt;
&lt;li&gt;A bot check shown to our crawler may not be shown to GPTBot or Googlebot, which firewall vendors can recognize by their published addresses. Our bot-check numbers show what a new, honestly identified automated visitor gets.&lt;/li&gt;
&lt;li&gt;Firewalls can also block a specific AI crawler at the network edge without any sign in robots.txt or in what our crawler saw. We couldn't measure that without pretending to be that crawler, which we don't do. So the true share of sites that aren't fully open to AI crawlers is, if anything, higher.&lt;/li&gt;
&lt;li&gt;Each site was checked once, on 22 October 2026. Sites change.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Data.&lt;/strong&gt; The totals by category, without any site names, are available to &lt;a href="https://www.preferly.ai/research/ai-crawler-study.csv" rel="noopener noreferrer"&gt;download as CSV&lt;/a&gt;. The aggregate data is free to reuse under the &lt;a href="https://creativecommons.org/licenses/by/4.0/" rel="noopener noreferrer"&gt;Creative Commons Attribution 4.0 International license (CC BY 4.0)&lt;/a&gt;. Please credit: Preferly, "How many small business websites block AI crawlers?" (2026), &lt;a href="https://www.preferly.ai/research/ai-crawler-study" rel="noopener noreferrer"&gt;https://www.preferly.ai/research/ai-crawler-study&lt;/a&gt;. Sample drawn from OpenStreetMap data, © OpenStreetMap contributors, ODbL.&lt;/p&gt;

&lt;p&gt;If you want to see what AI crawlers get from a specific site, Preferly's &lt;a href="https://www.preferly.ai/ai-crawler-check" rel="noopener noreferrer"&gt;free AI crawler check&lt;/a&gt; runs the same two requests used in this study and does not ask for an email.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>seo</category>
      <category>webdev</category>
      <category>datascience</category>
    </item>
  </channel>
</rss>
