Every week another AI crawler shows up in your logs: ClaudeBot, GPTBot, OAI-SearchBot, PerplexityBot, Google-Extended. Your robots.txt might allow all of them — and your CDN might still be answering with a 403 challenge page before the crawler ever sees your content. Nobody checks for that contradiction, because it means actually making the requests, not just parsing a text file.
So I built AgentReady: you give it a URL, it makes six real HTTP requests — one as a normal browser, five using the actual published user-agents of ClaudeBot, GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended — plus checks for robots.txt, llms.txt, llms-full.txt, sitemap.xml and security.txt. It's live-fetched from our own server (no domain sends CORS headers to a site it's never heard of), so there's no caching or guessing.
What it actually measures, in order of weight:
- User-agent parity — status code and content length each crawler actually got back, versus what a browser got. Served / challenged / thin.
- Text present without JavaScript execution
-
robots.txtat the root, per-agent, including Cloudflare's Content-Signal header - JSON-LD and prices declared in markup, not just painted on screen
- Ratio of content tokens to total tokens inside
<main> - Discovery files and basic semantic structure
Output is a 0-100 score with fixes ordered by points recoverable, not just a checklist.
I ran it against a few well-known sites: nytimes.com came back a D (58) — 4 out of 5 AI crawlers got a 403 straight from the edge, and robots.txt disallows everything. stripe.com scored A+ (93).
Worth being upfront: this space is crowded. There are a bunch of free robots.txt parsers out there, and crawlreadiness.com does something very close to this already (32 bots, real fetches, scored, paid tiers above a free daily limit). AgentReady is free, has no login, and the badge it generates is meant to sit on your own site — but I'm not claiming a moat here. What I actually think matters is the sibling project, The Agent Web Index: the same measurement, run continuously across the web, as a public index instead of a one-off check. That's the part nobody else is doing yet.
Curious what other checks people would want in the per-agent breakdown, or which crawlers I'm missing.
Top comments (0)