How to check if your site is blocking AI crawlers (robots.txt won't tell you)

We build LovedByAI, a generative engine optimization platform for small business WordPress sites, which means we run crawler-access checks on live sites all day. In August we pointed that at 314 of them, fetching each one as 38 different AI crawlers and comparing the result against four weeks of their server logs.

One finding is a genuine observability trap, and I have not seen it written up anywhere.

A site blocking OAI-SearchBot at the origin logged a page request from it 84.6% of the time. A site with no block at all logged one 87.3% of the time.

Three points apart. So if you are checking your own logs to find out whether AI crawlers can reach you, a working block and no block look the same.

Reading robots.txt does not help either, and that turns out to be the more common failure: of those 314 sites, 135 refused at least one AI crawler and only 23 had written a rule about it.

That is the gap, as our own crawlability view reports it: what robots.txt declares on one side, what the server actually answered on the other. A site can carry one of each at the same time, and only one of them is something a human ever typed.

Here is why that happens, and how to test it properly.

Edge vs origin: where an AI crawler block lives

A block at the edge happens before the request reaches your server. A block at the origin happens after. That single distinction decides everything about what you can observe.

Edge. Cloudflare, a CDN firewall rule, a WAF. The crawler's request is terminated at the network boundary. Your application never runs. Your access log never gets a line. The crawler gets a 403 and leaves.

Origin. A security plugin, an nginx map on $http_user_agent, an Apache RewriteCond. The request completes a TCP handshake, hits your server, gets logged, then gets refused.

So the origin-blocked request appears in your logs as a visit. Your crawler-stats plugin counts it. Your dashboard shows GPTBot stopping by.

It left with a 403 and nothing else.

Here's the same comparison for the edge case:

• OAI-SearchBot blocked at the origin: logged on 84.6% of sites
• OAI-SearchBot not blocked at all: logged on 87.3% of sites
• OAI-SearchBot blocked at the edge: logged on 0.0% of sites

The edge column is the only one that's legible. Zero requests, because nothing ever arrived. An edge block is loud in your logs by being completely silent. An origin block hides in plain sight.

Why robots.txt will not tell you if AI crawlers are blocked

The obvious instinct is to skip logs and read robots.txt. That's worse.

Of the 135 sites refusing a crawler, only 23 declared it. The other 112 refused at the server with nothing in the site's own config to show for it. The request arrives, the server says 403, and robots.txt still reads like a welcome mat.

And a written rule isn't reliable in the other direction either, because robots.txt is a convention, not an enforcement mechanism:

• 33.3% of sites disallowing GPTBot still logged real GPTBot page requests
• 17.6% of sites disallowing ClaudeBot still logged ClaudeBot

Neither figure counts robots.txt or sitemap fetches, which aren't ingested. Every request counted is a crawler asking for a page it was told not to take.

So robots.txt tells you what you meant. Your logs tell you what arrived. Neither tells you what the crawler got.

How to check if your site is blocking AI crawlers

Send the request yourself, as the crawler, and read the status code.

The part people skip is the control. If you fire ten requests at a site from one IP and get ten 403s, you've learned nothing. It could be your IP, a rate limiter, or the site being down. You need an unbranded request from the same address to prove the site is up and that the refusal is attributable to the user-agent string.

#!/usr/bin/env bash
SITE="${1:?usage: ./crawlcheck.sh https://example.com}"

code() { curl -s -o /dev/null -w "%{http_code}" -A "$1" --max-time 15 "$SITE"; }

CONTROL=$(code "crawlcheck-control/1.0")
echo "control          $CONTROL"
[ "$CONTROL" = "200" ] || { echo "control failed - results below are meaningless"; exit 1; }

for UA in \
  "GPTBot/1.2" \
  "OAI-SearchBot/1.0" \
  "ChatGPT-User/1.0" \
  "PerplexityBot/1.0" \
  "Claude-SearchBot/1.0" \
  "ClaudeBot/1.0" \
  "Bytespider" \
  "Amazonbot/0.1"
do
  printf '%-18s %s\n' "$(echo "$UA" | cut -d/ -f1)" "$(code "$UA")"
  sleep 1
done

200 means reachable. 403 means you found one.

That covers eight crawlers. There are 38 tokens worth checking, including consent switches like Google-Extended and Applebot-Extended that are not crawlers at all and can only be evaluated against robots.txt. Keeping that list current as vendors add and rename tokens is the tedious part. LovedByAI automates the check across all 38 with the unbranded control request built in, and names each crawler that gets refused.

That sleep 1 matters more than it looks. Which brings me to the thing that nearly broke our own numbers.

Why rate limiting breaks an AI crawler check

Our check fires around 30 requests at a site in quick succession. That is enough to trip a rate limiter on its own.

A 429 therefore tells you about request volume, not about how the site treats AI crawlers. It's a measurement artefact of the tool doing the measuring.

We excluded every rate-limit response from every figure in the study. Including them would have added 107 refusals across 39 sites and made the headline considerably bigger.

If you're building anything like this, space your requests out and throw away the 429s. Otherwise you're measuring your own burst.

What an AI crawler block looks like: 403, 406 and challenges

Across 558 refusals where robots.txt explicitly allowed the crawler:

• 509 were a plain 403 Forbidden, 93% of the total
• 24 were a Cloudflare browser challenge, which to a bot is an unsolvable puzzle and therefore a locked door
• 20 were 406 Not Acceptable, typically a user-agent rule at the web server
• 5 were a Cloudflare managed-rule 403

That 93% is the useful part. It's the signature of a rule keyed to the user-agent string and nothing else. Not behaviour, not rate, not fingerprinting. Just string matching against a list somebody pasted in years ago.

Which is exactly why this catches crawlers nobody meant to catch. A "block bad bots" toggle switched on in 2021 has no idea ChatGPT-User would exist in 2026.

Which AI crawlers actually crawl your site?

Once the control returns 200 and nothing returns 403, you know crawlers can reach you. The next question is which ones actually do, and that needs request logs rather than a probe.

The two answer different things. The probe tells you who can get in. The logs tell you who bothered, and in what proportion. Raw crawler totals are usually dominated by training crawlers like Meta-ExternalAgent, which will bury the assistant crawlers you care about if you read the totals without splitting them.

Training crawlers vs assistant crawlers: GPTBot vs OAI-SearchBot

Not all of these are the same thing, and the naming does nothing to help:

• Training crawlers. GPTBot, CCBot, Bytespider, Amazonbot, Google-Extended. They take content to train models. Blocking them is a legitimate, common position and costs you nothing in AI search.
• Assistant crawlers. OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, DuckAssistBot. They fetch a page because someone just asked a question. The response ends in a citation with your link in it.

GPTBot and OAI-SearchBot are both OpenAI. They do completely different jobs. A UA rule matching GPT catches one; a rule matching OpenAI or a blanket bot filter catches both.

In our sample, 102 of the 135 affected sites block only training crawlers, deliberately and consistently. 33 sites block at least one assistant crawler, and 31 of them never wrote it down anywhere. That's not a policy. That's a side effect.

Does blocking AI crawlers cost you AI traffic?

The thing you'd most want to know is whether blocked sites get less AI traffic. We can't show it.

Median AI referral visits, June to August: 12 for the 269 reachable sites, 6 for the 11 blocking OpenAI at the edge, 5 for the 17 blocking at the origin. Points the right way. Then the 17 sites disallowing OpenAI in robots.txt came in at 19, above the reachable group, which is the opposite.

Groups of 11 to 17 sites, no control for site size or niche. That's noise, and I'd rather say so than dress it up.

What this data does establish: a large minority of sites refuse AI crawlers, most of them can't see it from their own configuration, and the most common kind of block is the kind your logs can't distinguish from success.

---

Run the script against your own domain. It takes ten seconds. If the control returns 200 and anything else returns 403, you've found something you didn't know was there.

Method, full numbers and the ones we threw out: the 314-site study.

If you'd rather not maintain a token list by hand, LovedByAI is a generative engine optimization platform for small businesses built around WordPress. It automates the optimization work, checks all 38 crawler tokens with the control request built in, and reports AI mentions and AI referral traffic back. The crawler check is free to run.

First-party crawlability checks and bot logs. 314 websites, 38 crawler tokens, 27 to 30 August 2026, validated against logs from 1 to 26 August. No site identified.

Comments

[Dmitry Sergeev]

lol that test with 314 sites is wild, anyone know