TLDR We sent the same homepage request to 2,096 sites twice, once as Chrome and once as GPTBot, from one US datacenter IP. A "block" turned out to be a 403, a 402 payment wall, a challenge page or a 429, and which one a crawler gets depends on the CDN in front of the site. The classifier and the pairing code below are rebuilt from that study's scripts and trimmed to run on any domain list.
For the AI Crawler Block Index we measured how 10,894 sites treat AI crawlers, on paper in robots.txt and in practice when the crawler shows up. The practice half ran on 2,096 of them, the top 1,000 open-web domains plus every news publisher in the sample. Each site got two requests from the same exit IP. One carried a standard Chrome user agent and the other carried OpenAI's published GPTBot string.
On publisher sites the browser got a page 83.8% of the time and GPTBot got one 54.2% of the time. The IP and the URL were identical, so the user agent string was the only difference the sites could react to.
Send the same request twice
The only dependency is requests. The browser string is a recent Chrome on macOS, and the GPTBot string is the one OpenAI documents.
import requests
UAS = {
"browser": ("Mozilla/5.0 (Macintosh; Intel Mac OS X 15_8_1) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36"),
"gptbot": ("Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; "
"GPTBot/1.2; +https://openai.com/gptbot)"),
}
def fetch(url, ua, timeout=15):
try:
r = requests.get(url, headers={"User-Agent": ua, "Accept": "*/*"},
timeout=timeout, allow_redirects=True)
except requests.RequestException:
return None, "", {}
return r.status_code, r.text[:2000], r.headers
The fetch() function returns the status, the first 2,000 characters of the body and the response headers, which is everything the classifier and the CDN check read.
Chrome's request is there for comparison. When Chrome gets a page and GPTBot doesn't, the site refused the name. When both fail, the problem is the IP or the network, and the pair says nothing about AI policy. The same limit applies to anyone repeating this. Your requests carry GPTBot's string but don't come from OpenAI's IP ranges, so a site that verifies crawlers by IP treats them as impostors.
Turn a status code into an outcome
Each response goes into exactly one bucket. Challenge and CAPTCHA markers win over the status code, because both kinds of page arrive with a 200 or a 403.
CAPTCHA_MARKERS = ("hcaptcha", "recaptcha")
CHALLENGE_MARKERS = ("just a moment",)
def classify(status, body, headers=None):
if status is None:
return "error"
if (headers or {}).get("cf-mitigated") == "challenge":
return "challenge"
if status in (200, 206, 401, 403):
text = body.lower()
if any(marker in text for marker in CAPTCHA_MARKERS):
return "captcha"
if any(marker in text for marker in CHALLENGE_MARKERS):
return "challenge"
if status in (200, 206):
return "ok"
if status in (401, 403, 451):
return "blocked"
if status == 402:
return "blocked-payment"
if status == 429:
return "ratelimited"
if status >= 500:
return "server_error"
return f"other-{status}"
The body check runs on 200 and 403 alike, because CAPTCHA pages sometimes arrive with a success status and Cloudflare serves its challenge pages with a 403. 402 gets its own bucket on purpose. Cloudflare's pay-per-crawl answers with 402 Payment Required when a crawler has no paid arrangement with the site, which makes it a price for access. Folding it into 403 throws that away.
Keep error out of every rate. It means the request never got an answer, and about 15% of our requests ended there with proxy-level failures. A dropped connection tells you nothing about the site. Challenges and CAPTCHAs stay in and get a line of their own, and on 8.2% of the sites at least one of the two requests hit one.
A JavaScript shell with no article text returns 200 and gets ok as if it were a real page, so if you need to know what a crawler can read, add a length or selector check on the body. Cloudflare also documents a cf-mitigated: challenge response header on its challenge pages, and the classifier checks it before the body, since a header is a firmer signal than page text. The CAPTCHA markers also match pages that only load a reCAPTCHA script for a login form, so check a few captcha rows by hand.
Tag the CDN from the same response
The browser's response already carries the headers that name the CDN, so tagging costs no extra request.
def detect_cdn(headers):
h = {k.lower(): v.lower() for k, v in headers.items()}
if "cf-ray" in h or h.get("server") == "cloudflare":
return "cloudflare"
if "x-amz-cf-id" in h or "cloudfront" in h.get("via", ""):
return "cloudfront"
if "x-fastly-request-id" in h or h.get("x-served-by", "").startswith("cache-"):
return "fastly"
return "other" if h else "unknown"
def probe_pair(domain):
url = f"https://{domain}/"
row = {"domain": domain}
for label, ua in UAS.items():
status, body, headers = fetch(url, ua)
row[label] = classify(status, body, headers)
row[f"{label}_status"] = status
if label == "browser":
row["cdn"] = detect_cdn(headers)
return row
The Cloudflare check is the study's own definition, a cf-ray header or server: cloudflare. The CloudFront and Fastly checks look for headers those CDNs add to responses. A site that strips vendor headers gets tagged other, so expect that bucket to be bigger than the true share of small providers. The tag is read from the browser's response, the one the site handled as ordinary traffic.
Compare robots.txt with what happened
A paired result means more next to what the site says in robots.txt, and reading the file takes one more request per domain.
def robots_txt(domain):
try:
r = requests.get(f"https://{domain}/robots.txt",
headers={"User-Agent": UAS["gptbot"]}, timeout=15)
except requests.RequestException:
return None
if r.status_code == 404:
return "" # no file means no rules
return r.text if r.status_code == 200 else None
def gptbot_disallowed(robots):
robots = robots.lstrip("\ufeff")
groups, agents, rules = [], [], []
for raw in robots.splitlines():
line = raw.split("#", 1)[0].strip()
if ":" not in line:
continue
field, value = (part.strip() for part in line.split(":", 1))
field = field.lower()
if field == "user-agent":
if rules: # a new group starts
groups.append((agents, rules))
agents, rules = [], []
agents.append(value.lower())
elif field in ("allow", "disallow"):
rules.append((field, value))
groups.append((agents, rules))
return any("gptbot" in a and ("disallow", "/") in r and ("allow", "/") not in r
for a, r in groups)
REFUSED = {"blocked", "blocked-payment", "challenge", "captcha", "ratelimited"}
def policy_vs_practice(row, robots):
if robots is None:
return "robots.txt unreadable"
if "error" in (row["browser"], row["gptbot"]):
return "no answer"
banned = gptbot_disallowed(robots)
if banned and row["gptbot"] == "ok":
return "paper-only"
if row["browser"] != "ok":
return "no control"
if not banned and row["gptbot"] in REFUSED:
return "silent enforcer"
if row["gptbot"] == "ok" or row["gptbot"] in REFUSED:
return "consistent"
return "unclear"
The parser counts a ban only when GPTBot's own group says Disallow: / and nothing in that group allows the root back. A User-agent: * block covers every crawler and says nothing about AI, so the study didn't count it, and neither does this code. Python's urllib.robotparser falls back to the * group when no group names the bot. That's correct for a polite crawler deciding what to fetch and wrong for counting AI bans.
The robots.txt request also goes out as GPTBot, since GPTBot is the client that has to read it. In July, 592 sites in our subset disallowed GPTBot and answered our request, and 234 of them (39.5%) served it a page anyway. Open-web sites ignored their own ban in 21 of 45 cases (46.7%) and publishers in 213 of 547 (38.9%). At 45 sites, one site moves the open-web share by 2.2 points. We found the opposite case on 115 sites, which said nothing about GPTBot in robots.txt and refused it at the firewall while the browser got through.
A missing file (404) counts as no rules. A file the server refuses to hand over gets its own label. Between our July and September runs, 125 Cloudflare sites that had served us robots.txt in July didn't serve it in September, 97 of them with a 403, and a crawler that can't read the file can't follow it.
Group the answers by CDN
The last block runs the pairs over a domain list and counts GPTBot's outcomes per CDN.
from collections import Counter, defaultdict
from concurrent.futures import ThreadPoolExecutor
def by_cdn(rows, min_n=20):
table = defaultdict(Counter)
for r in rows:
if r["gptbot"] == "error":
continue # a network failure says nothing about the site
table[r["cdn"]][r["gptbot"]] += 1
for cdn, counts in sorted(table.items(), key=lambda kv: -sum(kv[1].values())):
n = sum(counts.values())
if n < min_n:
continue
shares = {cls: round(100 * k / n, 1) for cls, k in counts.most_common()}
print(f"{cdn:<11} n={n:<5} {shares}")
if __name__ == "__main__":
with open("domains.txt", encoding="utf-8") as f:
domains = [line.strip() for line in f if line.strip()]
with ThreadPoolExecutor(max_workers=4) as pool:
rows = list(pool.map(probe_pair, domains))
for row in rows:
row["policy"] = policy_vs_practice(row, robots_txt(row["domain"]))
by_cdn(rows)
print(Counter(row["policy"] for row in rows))
The study used the same floor of 20 rows per CDN. Below that, one site moves a bucket by more than five points. Each worker handles one domain at a time, so no site sees two of your requests at once. The study's shares keep network failures in the denominator and by_cdn() drops them, so the same sites give you shares up to 4 points higher.
On our 2,096 sites the table came out differently per vendor. Fastly served GPTBot 27.3% of the time and refused 61.5%, and 197 of those 214 refusals were 402s. Cloudflare served 51.3%, refused 16.9% and challenged 24.7%. CloudFront served 78.5% and refused 14.4%. Of the three, only Cloudflare answered with challenges.
A scraper that handles a 403 but can't run a JavaScript challenge sees its pass rate on one site list move by tens of points depending on which of these is in front. Log the class and the CDN next to the status code on every request, and that difference shows up in your first report.
The full study adds the robots.txt side for all 10,894 sites and the September re-run after Cloudflare changed its AI defaults. The aggregate tables, the 1,500 residential probes from the per-bot test and the scripts behind them are in the dataset repository under CC BY 4.0. The per-site datacenter probes behind the numbers above aren't in it.
Top comments (0)