DEV Community: LovedByAI

How to check if your site is blocking AI crawlers (robots.txt won't tell you)

LovedByAI — Wed, 23 Sep 2026 20:32:49 +0000

We build LovedByAI, a generative engine optimization platform for small business WordPress sites, which means we run crawler-access checks on live sites all day. In August we pointed that at 314 of them, fetching each one as 38 different AI crawlers and comparing the result against four weeks of their server logs.

One finding is a genuine observability trap, and I have not seen it written up anywhere.

A site blocking OAI-SearchBot at the origin logged a page request from it 84.6% of the time. A site with no block at all logged one 87.3% of the time.

Three points apart. So if you are checking your own logs to find out whether AI crawlers can reach you, a working block and no block look the same.

Reading robots.txt does not help either, and that turns out to be the more common failure: of those 314 sites, 135 refused at least one AI crawler and only 23 had written a rule about it.

That is the gap, as our own crawlability view reports it: what robots.txt declares on one side, what the server actually answered on the other. A site can carry one of each at the same time, and only one of them is something a human ever typed.

Here is why that happens, and how to test it properly.

Edge vs origin: where an AI crawler block lives

A block at the edge happens before the request reaches your server. A block at the origin happens after. That single distinction decides everything about what you can observe.

Edge. Cloudflare, a CDN firewall rule, a WAF. The crawler's request is terminated at the network boundary. Your application never runs. Your access log never gets a line. The crawler gets a 403 and leaves.

Origin. A security plugin, an nginx map on $http_user_agent, an Apache RewriteCond. The request completes a TCP handshake, hits your server, gets logged, then gets refused.

So the origin-blocked request appears in your logs as a visit. Your crawler-stats plugin counts it. Your dashboard shows GPTBot stopping by.

It left with a 403 and nothing else.

Here's the same comparison for the edge case:

OAI-SearchBot blocked at the origin: logged on 84.6% of sites
OAI-SearchBot not blocked at all: logged on 87.3% of sites
OAI-SearchBot blocked at the edge: logged on 0.0% of sites

The edge column is the only one that's legible. Zero requests, because nothing ever arrived. An edge block is loud in your logs by being completely silent. An origin block hides in plain sight.

Why robots.txt will not tell you if AI crawlers are blocked

The obvious instinct is to skip logs and read robots.txt. That's worse.

Of the 135 sites refusing a crawler, only 23 declared it. The other 112 refused at the server with nothing in the site's own config to show for it. The request arrives, the server says 403, and robots.txt still reads like a welcome mat.

And a written rule isn't reliable in the other direction either, because robots.txt is a convention, not an enforcement mechanism:

33.3% of sites disallowing GPTBot still logged real GPTBot page requests
17.6% of sites disallowing ClaudeBot still logged ClaudeBot

Neither figure counts robots.txt or sitemap fetches, which aren't ingested. Every request counted is a crawler asking for a page it was told not to take.

So robots.txt tells you what you meant. Your logs tell you what arrived. Neither tells you what the crawler got.

How to check if your site is blocking AI crawlers

Send the request yourself, as the crawler, and read the status code.

The part people skip is the control. If you fire ten requests at a site from one IP and get ten 403s, you've learned nothing. It could be your IP, a rate limiter, or the site being down. You need an unbranded request from the same address to prove the site is up and that the refusal is attributable to the user-agent string.

#!/usr/bin/env bash
SITE="${1:?usage: ./crawlcheck.sh https://example.com}"

code() { curl -s -o /dev/null -w "%{http_code}" -A "$1" --max-time 15 "$SITE"; }

CONTROL=$(code "crawlcheck-control/1.0")
echo "control          $CONTROL"
[ "$CONTROL" = "200" ] || { echo "control failed - results below are meaningless"; exit 1; }

for UA in \
  "GPTBot/1.2" \
  "OAI-SearchBot/1.0" \
  "ChatGPT-User/1.0" \
  "PerplexityBot/1.0" \
  "Claude-SearchBot/1.0" \
  "ClaudeBot/1.0" \
  "Bytespider" \
  "Amazonbot/0.1"
do
  printf '%-18s %s\n' "$(echo "$UA" | cut -d/ -f1)" "$(code "$UA")"
  sleep 1
done

200 means reachable. 403 means you found one.

That covers eight crawlers. There are 38 tokens worth checking, including consent switches like Google-Extended and Applebot-Extended that are not crawlers at all and can only be evaluated against robots.txt. Keeping that list current as vendors add and rename tokens is the tedious part. LovedByAI automates the check across all 38 with the unbranded control request built in, and names each crawler that gets refused.

That sleep 1 matters more than it looks. Which brings me to the thing that nearly broke our own numbers.

Why rate limiting breaks an AI crawler check

Our check fires around 30 requests at a site in quick succession. That is enough to trip a rate limiter on its own.

A 429 therefore tells you about request volume, not about how the site treats AI crawlers. It's a measurement artefact of the tool doing the measuring.

We excluded every rate-limit response from every figure in the study. Including them would have added 107 refusals across 39 sites and made the headline considerably bigger.

If you're building anything like this, space your requests out and throw away the 429s. Otherwise you're measuring your own burst.

What an AI crawler block looks like: 403, 406 and challenges

Across 558 refusals where robots.txt explicitly allowed the crawler:

509 were a plain 403 Forbidden, 93% of the total
24 were a Cloudflare browser challenge, which to a bot is an unsolvable puzzle and therefore a locked door
20 were 406 Not Acceptable, typically a user-agent rule at the web server
5 were a Cloudflare managed-rule 403

That 93% is the useful part. It's the signature of a rule keyed to the user-agent string and nothing else. Not behaviour, not rate, not fingerprinting. Just string matching against a list somebody pasted in years ago.

Which is exactly why this catches crawlers nobody meant to catch. A "block bad bots" toggle switched on in 2021 has no idea ChatGPT-User would exist in 2026.

Which AI crawlers actually crawl your site?

Once the control returns 200 and nothing returns 403, you know crawlers can reach you. The next question is which ones actually do, and that needs request logs rather than a probe.

The two answer different things. The probe tells you who can get in. The logs tell you who bothered, and in what proportion. Raw crawler totals are usually dominated by training crawlers like Meta-ExternalAgent, which will bury the assistant crawlers you care about if you read the totals without splitting them.

Training crawlers vs assistant crawlers: GPTBot vs OAI-SearchBot

Not all of these are the same thing, and the naming does nothing to help:

Training crawlers. GPTBot, CCBot, Bytespider, Amazonbot, Google-Extended. They take content to train models. Blocking them is a legitimate, common position and costs you nothing in AI search.
Assistant crawlers. OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, DuckAssistBot. They fetch a page because someone just asked a question. The response ends in a citation with your link in it.

GPTBot and OAI-SearchBot are both OpenAI. They do completely different jobs. A UA rule matching GPT catches one; a rule matching OpenAI or a blanket bot filter catches both.

In our sample, 102 of the 135 affected sites block only training crawlers, deliberately and consistently. 33 sites block at least one assistant crawler, and 31 of them never wrote it down anywhere. That's not a policy. That's a side effect.

Does blocking AI crawlers cost you AI traffic?

The thing you'd most want to know is whether blocked sites get less AI traffic. We can't show it.

Median AI referral visits, June to August: 12 for the 269 reachable sites, 6 for the 11 blocking OpenAI at the edge, 5 for the 17 blocking at the origin. Points the right way. Then the 17 sites disallowing OpenAI in robots.txt came in at 19, above the reachable group, which is the opposite.

Groups of 11 to 17 sites, no control for site size or niche. That's noise, and I'd rather say so than dress it up.

What this data does establish: a large minority of sites refuse AI crawlers, most of them can't see it from their own configuration, and the most common kind of block is the kind your logs can't distinguish from success.

Run the script against your own domain. It takes ten seconds. If the control returns 200 and anything else returns 403, you've found something you didn't know was there.

Method, full numbers and the ones we threw out: the 314-site study.

If you'd rather not maintain a token list by hand, LovedByAI is a generative engine optimization platform for small businesses built around WordPress. It automates the optimization work, checks all 38 crawler tokens with the control request built in, and reports AI mentions and AI referral traffic back. The crawler check is free to run.

First-party crawlability checks and bot logs. 314 websites, 38 crawler tokens, 27 to 30 August 2026, validated against logs from 1 to 26 August. No site identified.

WordPress SEO Plugins Weren’t Built for AI Crawlers. Here’s What Changes in 2026

LovedByAI — Mon, 14 Sep 2026 08:19:55 +0000

For years, the WordPress SEO stack has been pretty straightforward.

Install Yoast, Rank Math, AIOSEO, or something similar. Set your titles and descriptions. Generate a sitemap. Add schema. Fix your redirects. Make Google happy.

That still matters.

But ChatGPT, Gemini, Perplexity and other AI answer engines introduce a slightly different problem:

What does an AI crawler actually receive when it visits your page?

We recently compared five WordPress plugins through that lens:

Rank Math
Yoast
AIOSEO
LovedByAI
Squirrly

Full comparison of the best GEO plugins for WordPress

Disclosure before getting into it: I'm part of the team behind LovedByAI, so don't treat this as an independent ranking. What I think is more interesting than the ranking itself is what we found when comparing how these tools approach AI discovery.

SEO and GEO are solving different problems

Traditional SEO plugins are primarily designed around search engines indexing a page and returning a link.

You optimize things like:

Meta titles
Canonicals
XML sitemaps
Structured data
Redirects
Internal linking
Keyword usage

GEO, or Generative Engine Optimization, is trying to influence a different outcome.

Instead of:

"Can Google rank this URL?"

The question becomes:

"Can an LLM understand this page well enough to use the business as part of an answer?"

That means page structure, entity clarity and machine readability become much more important.

The architectural difference I found most interesting

Most WordPress SEO plugins are what I'd call passive optimization.

They improve the existing page.

Rank Math might inject schema. Yoast adds metadata. AIOSEO adds LocalBusiness information.

But when an AI crawler visits the site, it still receives the underlying WordPress page.

Depending on the site, that can mean a pretty messy combination of:

Theme HTML
Page-builder markup
Navigation
Popups
JavaScript
Tracking scripts
Widgets
Cookie banners
Actual content
More JavaScript

For a human browser, that's fine. The browser knows how to render it.

For a machine trying to quickly determine:

What does this company do?
Who is it for?
Where does it operate?
What services does it provide?
Why is it different?

there's potentially a lot of noise.

The alternative is an active rendering layer.

Instead of only modifying the original WordPress HTML, you create a machine-oriented representation of the same page.

Something closer to:

<h1>Acme Accounting</h1>

<p>
Acme Accounting provides bookkeeping and tax services
for small businesses in Austin, Texas.
</p>

<h2>What services does Acme Accounting offer?</h2>

<ul>
  <li>Bookkeeping</li>
  <li>Business tax preparation</li>
  <li>Payroll management</li>
</ul>

alongside structured entity data.

The information hasn't fundamentally changed.

The representation has.

That is the approach we took with LovedByAI's LLM view.

So do you replace Yoast or Rank Math?

I don't think so.

This was probably the biggest conclusion from the comparison.

A GEO plugin and an SEO plugin shouldn't necessarily compete for the same slot.

Rank Math is very good at things like:

redirects
sitemap management
schema
on-page SEO
404 monitoring

Yoast remains extremely strong for:

editorial workflows
readability
metadata
mature SEO infrastructure

AIOSEO has particularly useful features for local businesses.

Those problems don't disappear because ChatGPT exists.

The stack increasingly looks more like:

WordPress
    ↓
Traditional SEO layer
    ↓
Yoast / Rank Math / AIOSEO
    ↓
AI visibility layer
    ↓
Machine-readable content + entity data

In other words, SEO hygiene + GEO infrastructure, rather than SEO vs. GEO.

llms.txt has gained popularity, but it's not the right answer

Another thing worth mentioning.

There has been a lot of attention around llms.txt.

It's an emerging convention, but generating an llms.txt file doesn't suddenly make a site optimized for AI.

The actual page still matters.

An AI system needs to extract clear facts, relationships and answers from your content.

Things like:

clean semantic structure
descriptive headings
direct answers
Organization and Service schema
consistent entity information
crawlable content

are much harder to fake with one text file.

How I'd evaluate a WordPress GEO plugin

After working on this space, I think the useful questions are less about how many "AI features" appear on the pricing page.

I'd look at:

1. What does an AI crawler actually receive?

Inspect the output, not the marketing copy.

2. Does it improve machine readability or just generate content?

An AI writing assistant and an AI-search optimization layer aren't the same thing.

3. Does it produce useful structured data?

Not just generic Article schema.

4. Does it work alongside the existing SEO stack?

Replacing a mature SEO setup just to add GEO rarely makes sense.

5. How much work does it require after installation?

For SMBs especially, a dashboard containing 47 recommendations isn't necessarily a solution.

The bigger shift

The web has historically been optimized primarily for two consumers:

Humans
Googlebot

We're adding a third:

Humans
Search crawlers
AI crawlers / answer engines

And they don't necessarily want information presented in exactly the same way.

I suspect the WordPress stack will increasingly reflect that.

SEO plugins will continue handling technical search hygiene. A new layer of tools will focus specifically on making sites easy for AI systems to parse, understand and cite.

I also created a free GEO auditor that you can run and see the type of suggestions that we are talking about.