DEV Community: David Black

I scanned 1,902 local-business websites for AI readiness. One in four already has an llms.txt.

David Black — Mon, 12 Oct 2026 17:43:03 +0000

I build a tool that checks whether AI answer engines can read a website. On 12 October I pointed it at 1,902 ordinary local-business sites in the USA and UK: dentists, plumbers, hairdressers, vets, gyms, estate agents and five other trades, in twenty big cities. 1,355 of them could be read and scored.

I expected the story to be "small businesses are blocking AI and do not know it". That is not what the data says. Here is what it does say.

One in four already has an llms.txt, and most did not write it

338 of the 1,355 sites (24.9%) serve an llms.txt file. That is far more than I expected for businesses that mostly do not know the file exists.

The reason is the site builder:

Platform	Sites with an llms.txt
Wix	118 of 121 (97.5%)
Shopify	27 of 28 (96.4%)
Duda	19 of 19
WordPress	109 of 576 (18.9%)
Custom-built	38 of 342 (11.1%)
Squarespace	4 of 110 (3.6%)

If you are on Wix, you almost certainly have one. If you are on Squarespace, you almost certainly do not. Whether anyone chose that is another matter.

Almost nobody blocks AI crawlers in robots.txt

Only 9 sites (0.7%) have a robots.txt rule that shuts out an AI answer fetcher such as OAI-SearchBot or PerplexityBot. 14 (1.0%) shut out a training crawler such as GPTBot or ClaudeBot.

I counted a block only when the file names the crawler and disallows the whole site. By that measure, local businesses are not opting out of AI. They have not touched the file.

The real gap is that sites do not say who they are

This is the finding that matters most for the businesses themselves.

621 sites (45.8%) have no structured data describing the business at all. No name, address, phone number or hours in a form a machine can read without guessing.
796 (58.7%) do not give their opening hours in structured data.
Only 235 (17.3%) state every fact the scan looks for.
Custom-built sites are worst: 254 of 342 have none. On Wix it is 26 of 121.

An answer engine asked "who is open now near me" has to get that from somewhere. For most of these sites it is not the site.

Seven in ten have no questions and answers

970 sites (71.6%) had no FAQ page and no question-and-answer content on the pages read. Roofers and plumbers do best. Hairdressers and accountants do worst.

The typical score is 58 out of 100

The median readiness score is 58.1. A third of sites score under 50. By platform the medians run from 69.8 (Shopify) down to 47.7 (custom-built), which is mostly the structured-data gap again: builders add it for you, and a hand-built site only has what someone remembered to write.

One listed website in six no longer works

A side finding: 328 of the 1,902 sites I sampled from OpenStreetMap had no working website. 147 domains no longer resolve.

How it was done, and what it cannot tell you

The sample is independent businesses that list a website in OpenStreetMap, chains removed, up to 100 drawn at random per trade per country. Each site was read over plain HTTP, up to five pages, with no browser and no AI model. Twenty were checked by hand against a second reading and fifty were scanned twice.

Limits worth knowing before you quote any of this: five pages is not a whole site; the sample leans towards established premises in big cities; and the scan ran from one connection on one day. It says nothing about whether an AI engine actually recommends any of these businesses. It measures whether a site can be read and what it states, which comes first.

Every count, the method, the infographic and a CSV of the figures are in the full report: How ready are local-business websites for AI answers?

Contains data © OpenStreetMap contributors, available under the Open Database Licence.

Seeding a local directory with real listings on day one

David Black — Mon, 12 Oct 2026 15:06:43 +0000

Nobody adds their business to an empty directory, and nobody searches one twice. So a local directory has to open with listings already in it. Here is how to fill one with real businesses before launch, without inventing anything.

1. Count before you build

Pick the unit first: one trade across a set of cities, or every trade in one town. Then count how many real businesses exist in that unit. If "tree surgeons in ten cities" comes to 150 businesses, you have a directory. If it comes to twelve, you have a blog post.

A rough count takes an hour and saves you from building pages for cities that will hold one listing each.

2. Choose a source you are allowed to use

There are three honest ones.

Google Places API. The widest coverage, with phone numbers, websites and map positions. It is paid per request, and its terms limit what you may store and for how long, so read them before you design around it.
OpenStreetMap. Free and openly licensed (ODbL). You must credit it, and coverage is uneven: good for shops and clinics with premises, thin for trades that work from a van.
Public registers. Licensing boards, trade associations and council lists. Slow to collect, but often the only complete source for a regulated trade.

Copying another directory is not on the list. Their descriptions are their copyright, and their mistakes become yours.

3. Make one CSV the hand-off between every step

Whatever the source, land everything in one plain file with fixed column names. A workable minimum:

business_name,city,country,website,phone,street_address,postal_code,latitude,longitude,category_1,description

Only the name and the city are essential. The point of the file is that you can open it in a spreadsheet between steps, fix what is wrong and carry on. A pipeline you cannot inspect will import its errors at full speed.

4. Clean it before it goes anywhere

Remove duplicates. Match on the website's domain first, then on name plus city. The same business often appears under two spellings.
Decide about chains. A national chain with forty branches can swamp a niche directory. Keep them or drop them on purpose.
Drop the dead. A listing whose website no longer loads is usually a business that has closed.
Assign categories by rule. "Found by the search phrase tree removal" is a reliable reason to file a business under tree removal. A guess from the business name is not.

5. Write your own descriptions

Do not paste the text from each business's own site. It is theirs, and a page that repeats what is already on the web gives a search engine no reason to show yours. Write two or three plain sentences per listing from the facts you hold: what they do, where, and anything that sets them apart. If you use a language model for this, give it only those facts and check a sample by hand.

6. Fill in missing map positions

Rows from a places API arrive with latitude and longitude. Rows from a register do not. OpenStreetMap's Nominatim service will look addresses up for free, at no more than one request a second and with a user agent that says who you are.

7. Import as pending, then approve

Load the file with every listing held back, read a sample of fifty, and only then make them public. Skip duplicates on import so that fixing ten bad rows and loading the file again adds ten listings, not a second copy of everything.

Once they are live, give owners a way to claim and correct their listing. They know their opening hours better than any source you used.

What not to do

Do not publish a city page with nothing on it. Leave the city out until it has listings.
Do not invent businesses, reviews or ratings to make the place look busy.
Do not email the businesses you imported unless the law where they are allows it.

Where this comes from

I build DirectoryLab, a set of self-hosted directory kits. Its local kit runs these steps from the admin screens, and the page that explains how the data pipeline works, including which steps have to run on your own computer, is public whether or not you use the kit.

How to check whether your site blocks ChatGPT, Claude and Perplexity

David Black — Mon, 12 Oct 2026 14:50:16 +0000

Plenty of sites turn AI crawlers away without anyone having decided to. A security plugin, a CDN setting or a line copied into robots.txt years ago is enough. If an assistant cannot fetch your pages, it cannot quote them, and you will not hear about it.

Here is how to check, in about ten minutes.

Two kinds of crawler, and why the difference matters

AI companies run more than one crawler, and they do different jobs.

Answer fetchers read a page so an assistant can answer a question with it, often while someone is waiting:

OAI-SearchBot and ChatGPT-User (OpenAI)
PerplexityBot and Perplexity-User (Perplexity)
Claude-SearchBot and Claude-User (Anthropic)

Training crawlers collect pages to train future models:

GPTBot (OpenAI)
ClaudeBot (Anthropic)
CCBot (Common Crawl, whose data many labs use)
Google-Extended (a robots.txt token rather than a separate crawler; it controls use of your pages for Gemini)

You may reasonably want to block the second group and allow the first. Blocking everything with "AI" in the name is how a business ends up invisible in answers it would have liked to be in.

Step 1: read your robots.txt

Open https://yourdomain.com/robots.txt in a browser and look for any of the names above. This blocks one crawler from the whole site:

User-agent: OAI-SearchBot
Disallow: /

Things to know while reading it:

A crawler obeys the most specific group that names it. If there is a User-agent: GPTBot group, GPTBot ignores the User-agent: * group entirely.
Disallow: / blocks everything. Disallow: with nothing after it blocks nothing.
A crawler that is never mentioned falls back to the * group, so check that one too.

Step 2: check for a block that is not in robots.txt

This is the one people miss. Firewalls, bot-protection plugins and CDN "block AI bots" switches refuse requests before robots.txt is ever consulted. Your robots.txt can look fine while every AI request gets a 403.

You can get a rough signal by sending a request that identifies itself as a crawler:

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0)" \
  https://yourdomain.com/

A 200 is a good sign. A 403, 406 or 429 means something in front of your site is refusing that name.

Be honest about what this proves. You are not the real crawler, and some protection checks the IP address as well as the name. A refusal here is strong evidence of a block. A 200 is weaker evidence that you are open. Your server or CDN logs are the final word: search them for the crawler names and look at the status codes they actually received.

Step 3: decide, then write it down

If you want to be quoted in AI answers but not used for training, say so explicitly rather than relying on defaults:

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

Then find the setting that caused any firewall block and add the same exceptions there. Changing robots.txt does nothing if the request never gets that far.

What this does not do

Letting a crawler in means it can read you. It does not mean an assistant will recommend you. That depends on what your pages say and who else says it.

A shortcut for step 1

I build MentionMachine, which has a free robots.txt checker for AI crawlers. It reads your file, shows which crawlers are allowed, blocked or never mentioned, and gives you a patch to copy. It writes nothing to your site and needs no account.

What a niche directory needs before it can earn anything

David Black — Mon, 12 Oct 2026 07:45:23 +0000

Directory sites look easy: a list of businesses and a search box. The list is the easy part. I have built several of these and sold a few, and the things that decided whether a site earned were never on the listing page.

Here is what I would have in place before expecting any income.

1. Real data on day one

An empty directory is a dead one. Nobody claims a listing on a site with twelve entries. Seed it from a source with structured data. For local services that usually means a places API, which gives you a name, address, phone number, opening hours and categories for each business.

Decide your unit of content early: one page per business, one per city, and one per category in a city.

2. Pages that match what people search

People search for "window cleaners in Leeds", not for your home page. The pages that bring visitors are the category-in-a-city pages, so give each one its own introduction, a count of listings and links to nearby cities.

Do not publish a city page with nothing on it. Keep it out of the index until it has real listings, or you fill the site with thin pages.

3. A way for owners to claim and pay

The path is: find my business, claim it, verify by email, upgrade. Each extra step loses people, so keep it to one paid tier at first and send them to a hosted checkout rather than building your own card form.

4. More than one way to earn

In rough order of effort:

Featured listings. The simplest thing to sell, because the buyer can see what they get.
Leads. A quote form on each category page, with the enquiry forwarded or sold to businesses in that area.
Ads. Worth adding once category pages have steady traffic, and not before.

5. An admin you will actually use

You need a moderation queue, a way to edit and merge duplicates, refunds and a log of who changed what. If fixing a listing takes a database query, you will stop fixing listings.

6. Automation for the boring parts

Scheduled jobs should refresh hours and closures, rebuild the sitemap, send receipts and email you a daily summary. A directory that needs you every day is a job, not an asset.

What I would skip at the start

User reviews, because of the moderation load. A mobile app. A second niche. Get one niche earning first.

Where this comes from

I packaged the engines my own directories run on as DirectoryLab. The setup guides are free to read and cover each of the steps above, whether or not you use the kits.

How to write an llms.txt file for a small business site

David Black — Mon, 12 Oct 2026 07:34:53 +0000

AI assistants now answer questions like "who is the best plumber near me" by reading websites. An llms.txt file is a small, plain-text page at the root of your site that tells them who you are and which pages to read first.

It is an emerging convention, not a standard, and no engine has promised to use it. It takes about ten minutes, so I add one to every site I build.

What goes in it

The format is Markdown with four parts, in this order:

An H1 with the name of the business or site.
A blockquote with a one-line description.
One or more H2 sections.
Under each section, a list of links, each with a short note.

# Harbour Plumbing

> Emergency and planned plumbing for homes in Bristol, open seven days a week.

## Services

- [Boiler repair](https://example.com/boiler-repair): Same-day repairs for gas boilers, with fixed call-out prices.
- [Bathroom fitting](https://example.com/bathrooms): Full bathroom installs, from design to tiling.

## About

- [Pricing](https://example.com/pricing): Call-out fees and hourly rates.
- [Contact](https://example.com/contact): Phone, email and service area.

Rules I follow

Use absolute URLs. A crawler reading the file on its own has no base URL to resolve against.
Write each note as a fact, not a slogan. "Same-day repairs for gas boilers" tells a model something. "Best-in-class solutions" tells it nothing.
List the pages a customer would ask about. Services, prices, service area and contact. Leave out legal pages and tag archives.
Keep it short. Twenty good links beat two hundred.
Serve it as plain text at https://yourdomain.com/llms.txt, with a 200 status.

Check your robots.txt at the same time

An llms.txt is no use if the crawlers are blocked. Look for rules that disallow GPTBot, ClaudeBot or PerplexityBot. Some security plugins and CDN settings add these for you, so it is worth reading the file your site actually serves.

What it does not do

A good llms.txt helps an engine read your site. It does not make the engine recommend you. The only way to know whether you are mentioned is to ask the engines the questions your customers ask, and read the answers.

A shortcut

I build MentionMachine, which has a free llms.txt generator. It builds the file from what you type and can validate one you have already published. There is no account, and nothing is rewritten by a model. There is also a robots.txt checker for AI crawlers.

If you have added an llms.txt to your own site, I would like to hear whether you saw any change.