I build a small tool that checks whether AI models can actually reach a website. A few weeks ago I ran it against our own domain and found we were blocking one of the crawlers we care about, in our own robots.txt. Good reminder that this stuff breaks quietly.
Here's the part worth knowing even if you never touch a tool for it.
If your robots.txt blocks an AI crawler, that model won't cite you. Not "less often." Zero. It's a binary gate, and no amount of schema markup, FAQ blocks, or content tweaks changes it. The bot never fetched the page, so there's nothing to quote.
The trap is that you can block one without meaning to. A lot of the "block the AI bots" advice from 2023-2024 got pasted into robots.txt files and then forgotten. Some CDNs and WordPress security plugins add rules of their own. So the owner assumes the site is open, and one specific model has been locked out for a year.
How to check it yourself, no tool needed:
- Open yoursite.com/robots.txt in a browser.
- Look for User-agent lines matching the AI crawlers. The ones that matter right now:
- GPTBot and OAI-SearchBot (OpenAI / ChatGPT)
- ClaudeBot (Anthropic / Claude)
- Google-Extended (Gemini)
- PerplexityBot
- If there's a "Disallow: /" under any of those, that model is locked out. If they aren't mentioned at all, you're open to them by default, which is fine.
That's the whole check for the part that's genuinely black and white.
Now the honest caveat, because I'd rather you trust the rest of this. Getting crawled is necessary, not sufficient. It lets you be cited. It doesn't make you cited.
We ran our on-site "readiness" score against real citations across 44 domains. The correlation was basically nothing (Pearson around -0.08). What predicted citations wasn't markup or content depth, it was brand prominence. Well-known domains got cited, everyone else mostly got zero, no matter how clean their SEO was. Small sample, one model, one snapshot, so don't treat it as a study. But it made me skeptical of every tool that promises "optimize on-page, get cited" (including the pull to oversell my own). The crawl gate is real. The rest is mostly being someone the model already trusts.
So the practical takeaway is small and boring: make sure you aren't accidentally locked out. That's the one move with a guaranteed effect. Everything after it is a slower game.
If you'd rather not read robots.txt by hand across a pile of sites, the CLI is open source (causabi-geo on PyPI) and there's a hosted version at causabi.com. Either way, happy to answer questions in the comments.
Top comments (1)
didn't even realize robots.txt could be this tricky with the different ai agents, thanks for the heads up