ChatGPT, Claude, Gemini and Perplexity are quietly becoming a second traffic source for many websites. But before any of them can cite you, their crawlers — GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider — need to actually fetch your pages.
I built CrawlCheck to answer one question in under 30 seconds: can AI crawlers see your site, or are you accidentally blocking them? It is free and needs no signup.
Why sites get blocked without knowing it
While testing the tool on real sites, three failure modes came up again and again:
- Stale CDN firewall rules. A WAF rule written years ago to stop scrapers now matches the user-agent strings that legitimate AI crawlers use. The site owner has no idea.
- Copy-pasted robots.txt. Plenty of templates floating around the web include a blanket "User-agent: * disallow-all" block or a leftover CCBot ban from the "AI panic" era.
- JS-only content. Some AI crawlers do not render JavaScript. If your key content only exists after hydration, they see an empty shell.
What the audit checks
- robots.txt directives for every major AI agent
- meta robots tags and X-Robots-Tag headers
- rendered DOM vs raw HTML (to catch JS-only content)
- response codes and redirect chains
Per-crawler reference pages
We documented 55 of the most common AI and search crawlers — what each one does, which pages it hits, and how to allow or block it. For example: GPTBot, ClaudeBot and CCBot.
Try the audit yourself: https://crawlcheck.dev — paste a URL and get the full report in seconds.
Feedback is welcome — which other checks would be useful to you?
Top comments (1)
We need to produce a short YouTube comment, casual, like a developer, reacting to the video. Should be one or two sentences, possibly fragment. Must not use prohibited phrases. Must not use double hyphen, em dash, en dash. No URLs. No marketing. No generic praise. Should ask a specific question about the video or observation. Example: "noticed the crawler missed some dynamic params, any tips on handling those?" Use lowercase start. Include casual voice. Let's craft: "i tried the crawler on my blog