With AI crawlers like GPTBot (OpenAI), ClaudeBot (Anthropic), and Google-Extended (Gemini) hitting websites daily, most site owners
have no idea what's happening.
I built an AI Readiness Checker that scores your website on five categories.
## 1. robots.txt AI Bot Rules
Are you explicitly allowing or blocking AI crawlers? Most sites have a generic User-agent: * / Allow: / and call it done. That
means every AI bot can freely scrape your content for training data.
The tool checks rules for 10 specific AI bots:
- GPTBot & ChatGPT-User (OpenAI)
- ClaudeBot & Claude-Web (Anthropic)
- Google-Extended (Gemini training)
- PerplexityBot
- Bytespider (ByteDance)
- CCBot (Common Crawl)
- FacebookBot
- Applebot-Extended
## 2. llms.txt
This is a new standard (llmstxt.org) — a plain text file at your root that tells AI systems what your site is
about. Think of it as a README for AI crawlers. Almost nobody has one yet.
## 3. Structured Data
JSON-LD, Open Graph tags, Twitter Cards. These help AI systems understand your content structure and cite it correctly.
## 4. Content Citability
Do you have clear headings, lists, and enough textual content for AI to cite? A page with just an interactive tool and no text scores
low.
## 5. AI Meta Directives
The noai and noimageai meta directives let you opt out of AI training on a per-page basis.
## Try it
Check your site's AI readiness score — it takes about 5 seconds and you will probably
be surprised by the result.
Also: if you need to build a robots.txt with AI bot rules, there is a visual robots.txt
generator with one-click presets for blocking AI crawlers.
If you want to dig deeper, these guides go into each topic:
Top comments (1)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.