AI Bots Are Everywhere Now
GPTBot, ClaudeBot, Applebot — every major AI company now has a web crawler. And websites are scrambling to block them.
But how do you know if YOUR robots.txt is actually working? Most robots.txt checkers only test Googlebot. They don't tell you if GPTBot is blocked, or if your Allow/Disallow rules have priority conflicts.
What I Built
Robots.txt Checker — a free tool that:
- Fetches any site's robots.txt and parses every rule
- Tests specific paths against specific user-agents
- Supports AI bot testing — GPTBot, ChatGPT-User, ClaudeBot built into the dropdown
- Shows sitemaps found in the file
- Displays raw content for manual inspection
Try It: Check google.com
Paste google.com and you'll see:
- 4 user-agent groups with dozens of Disallow rules
- Sitemap references
- Which paths are blocked for Googlebot vs other crawlers
The AI Bot Test
Select GPTBot from the dropdown, enter a path like /, and see if the site blocks OpenAI's crawler. Many major sites now have:
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
How the Path Testing Works
Robots.txt has a longest-match-wins rule:
User-agent: *
Disallow: /admin/
Allow: /admin/public/
If you test /admin/public/page, the Allow rule wins because it's a longer match. My checker implements this correctly — many tools don't.
Why This Can't Be Replaced by ChatGPT
ChatGPT can explain robots.txt syntax. But it can't:
- Fetch a live robots.txt from any domain
- Parse real-world edge cases (comments, whitespace, malformed entries)
- Test your specific URL against your specific rules
- Tell you what a site's robots.txt looks like RIGHT NOW
You need an actual HTTP request to get actual data.
Try It
robotschecker.vercel.app — free, no signup.
Check if your site is blocking AI crawlers. Check if competitors are blocking crawlers. Verify your SEO team's robots.txt changes actually work.
Part of the DevTools Hub collection — 35+ free developer tools.
Top comments (0)