Disclosure: we build Crawl Readiness, a website check that also sells paid audits and monitoring. We also maintain Machine Readiness, the standard Crawl Readiness measures against. No standards body or AI company has adopted it. Both are made by Tundrastone (Ontario, Canada).
On 1 November 2026 we checked the top 1,000 hostnames of Tranco list 8PXYV for edition 1 of the monthly AI Access Index; 684 answered. Of the 547 tested sites that allow at least one of four AI crawlers in robots.txt (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot), 62 (11.3%) refused at least one crawler they allow, with a 401, 403, 429 or 503. Leaving out the 17 sites whose only refusals were 429s, it is 45 (8.2%). Of the 124 tested sites that block at least one of the four, 79 (63.7%) answered at least one crawler they block with a 2xx status.
Our test sends a user-agent string that names each crawler, from a cloud runner, not the vendors' IP addresses, so a site that verifies crawlers by address could refuse the test and admit the vendor's own crawler. The 11.3% counts sites that turned away a test request, so it is more exposed to that than the 63.7% is. A page served to a user-agent string does not show what the vendor's crawler would get.
robots.txt is a request, not a lock: it asks, and only the server, or a CDN or firewall in front of it, can refuse. A 2xx for a crawler that robots.txt asks to stay away breaks no rule, so we call it served, not a fault. In the standard we maintain, server enforcement is optional (A5, a MAY). Open or closed, a policy is a choice, and a block that holds is reported as working as intended.
Method
- Sample. Tranco list 8PXYV, top 1,000, downloaded on 2026-10-11 and frozen. 684 answered; 316 did not (some are infrastructure hostnames with no website). 599 of the 684 could be tested, meaning the server's answers can be read against its robots.txt; 85 could not.
- Run. 13:27 to 13:37 UTC: each site once, six at a time, nothing retried.
- Server test. The check's own request is the baseline. The homepage is then requested again as each of the four crawlers, keeping only the status. A crawler counts as blocked when robots.txt disallows it; a site with no robots.txt allows all four.
Recount the figures from the CSV
Save results.csv and this script as recount.mjs, then run node recount.mjs results.csv:
// split(",") is safe: only the last column holds commas.
import { readFileSync } from "node:fs";
const [head, ...lines] = readFileSync(process.argv[2], "utf8").trim().split("\n").map((l) => l.split(","));
const rows = lines.map((l) => Object.fromEntries(head.map((h, i) => [h, l[i]])));
const four = ["GPTBot", "OAI-SearchBot", "ClaudeBot", "PerplexityBot"];
const has = (r, key, value) => four.some((c) => r[c + key] === value);
const answered = rows.filter((r) => r.status === "ok");
const tested = answered.filter((r) => r.integrity !== "unknown");
const allows = tested.filter((r) => has(r, "_robots", "allowed"));
const blocks = tested.filter((r) => has(r, "_robots", "blocked"));
const refused = allows.filter((r) => has(r, "_outcome", "refused-while-allowed"));
const refusedNot429 = refused.filter((r) => four.some((c) => r[c + "_outcome"] === "refused-while-allowed" && r[c + "_status"] !== "429"));
const served = blocks.filter((r) => has(r, "_outcome", "served-while-blocked"));
const opened = blocks.filter((r) => r.homepage_status.startsWith("2"));
console.log("answered", answered.length, "tested", tested.length);
console.log("allow base", allows.length, "refused", refused.length, "without 429-only", refusedNot429.length);
console.log("block base", blocks.length, "served", served.length);
console.log("own request opened", opened.length, "served", opened.filter((r) => served.includes(r)).length);
It prints 684 and 599, then 547, 62 and 45, then 124 and 79, then 91 and 74. In the integrity column, both (2 sites) counts in each, so 77 + 2 = 79 and 60 + 2 = 62. In summary.json the headline counts are under server. Each row also has a score and a grade. A site whose robots.txt blocks AI crawlers in a pattern the check reads as deliberate is graded Restricted by Design, a choice rather than a fault. Its blocked crawlers still cost points, so its score falls.
Limits
- One pass, homepages only, from one vantage point. A 429 can be a rate limit, so the lower figure leaves 429-only refusals out.
- Served means a 2xx status and nothing more, so a 200 challenge page counts. Among tested sites that block at least one crawler and whose homepage opened for the check's own request, 74 of 91 (81.3%) answered a blocked crawler with a 2xx, against 79 of 124 above.
- This is edition 1, so there is no trend. In October we posted a study and said we would re-run it monthly. The Index is a new series, not that re-run: the study used a browser request as its baseline and a different network, so we do not set the two side by side. Edition 2 is scheduled for 1 December 2026.
- HasData published a declared-versus-served test of GPTBot in its AI Crawler Block Index, a different index with a similar name, using a July 2026 baseline and one US datacenter IP. Cloudflare describes the same gap between a robots.txt and its edge rules, and TollBit reports on bots that bypass disallow rules. Methods differ, so we do not compare the numbers.
Data and licences
- results.csv, changes.csv (a header row only in edition 1) and summary.json are CC BY-NC 4.0. Quote the numbers anywhere with credit, press included. Commercial use of the files, such as in a product or a dataset you sell, needs permission: support@crawlreadiness.com. Cite as "Crawl Readiness, AI Access Index, November 2026, www.crawlreadiness.com/ai-access-index/2026-11."
- The Index page said CC BY 4.0 from 1 November until the change merged at 16:48 UTC on 3 November, about 51 hours.
- The
rankcolumn is Tranco's; check Tranco's list page before reuse. - The crawler registry and the October study data are CC BY 4.0.
Links
Crawl Readiness checks that a website is set up for AI the way its owner intends: readable by the AI crawlers it wants reading it, closed to the ones it wants closed, and with a server that keeps to what its robots.txt says. Readiness is not a promise of being mentioned. Corrections: comment here or write to support@crawlreadiness.com.
- The AI Access Index, November 2026: https://www.crawlreadiness.com/ai-access-index/2026-11
- The standard Crawl Readiness measures against, Machine Readiness 1.1: https://machinereadiness.com/standard
- The free check, no signup: https://www.crawlreadiness.com/
Top comments (0)