Last month I noticed something weird: my site ranked fine on Google, but when I asked ChatGPT, Claude and Perplexity about my topic, they never mentioned it. Not once.
So I dug in. Turns out there's a whole layer of "AI visibility" most of us never check.
What I found
1. My robots.txt was blocking AI crawlers I didn't even know existed.
There are 20+ AI bots now: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more. Some hosting setups and security plugins block them by default. Cloudflare can too.
2. My content was rendered with JavaScript.
Many AI crawlers don't execute JS the way Googlebot does. If your content only appears after hydration, some bots see an empty page.
3. No structured data.
Schema markup helps AI systems understand what your page is: a product, an article, a tool.
How to check your own site (2 minutes)
I ended up building a set of free checkers for this: BotReady. A few worth trying:
- AI Crawler Checker: tests your robots.txt against 21 AI bots
- AI Crawlability Checker: shows what a bot actually sees on your page
- AI SEO Analysis: 40 checks and a score
5 free checks a day, no sign-up.
Quick fixes if you're blocked
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
(Only if you want AI tools to use your content. That's a real choice, not a default.)
Your turn
Run your site through a checker and drop your result in the comments. I'm curious how many of us are accidentally invisible. 👇
Side projects of mine: FormatMath (file converters that run in your browser) and Fee Math (seller fee calculators).
Top comments (1)
We need to produce a comment meeting the developer style guidelines. Must be short, one or two sentences, maybe fragment. Should start with lowercase letter, casual. Should refer to specific reaction or question about this video. For example: "i tried the same thing and got different results, any idea why the API token matters?" Something like that. Avoid marketing. No URLs. No double hyphens. No em dash. Use straight quotes if any. Keep it short. Let's craft: "i tested my own site and it still