Your site might be blocking AI crawlers by accident. Here's how to check.

Mikhail Shadrin — Thu, 30 Jul 2026 19:06:56 +0000

I build a small tool that checks whether AI models can actually reach a website. A few weeks ago I ran it against our own domain and found we were blocking one of the crawlers we care about, in our own robots.txt. Good reminder that this stuff breaks quietly.

Here's the part worth knowing even if you never touch a tool for it.

If your robots.txt blocks an AI crawler, that model won't cite you. Not "less often." Zero. It's a binary gate, and no amount of schema markup, FAQ blocks, or content tweaks changes it. The bot never fetched the page, so there's nothing to quote.

The trap is that you can block one without meaning to. A lot of the "block the AI bots" advice from 2023-2024 got pasted into robots.txt files and then forgotten. Some CDNs and WordPress security plugins add rules of their own. So the owner assumes the site is open, and one specific model has been locked out for a year.

How to check it yourself, no tool needed:

Open yoursite.com/robots.txt in a browser.
Look for User-agent lines matching the AI crawlers. The ones that matter right now:
- GPTBot and OAI-SearchBot (OpenAI / ChatGPT)
- ClaudeBot (Anthropic / Claude)
- Google-Extended (Gemini)
- PerplexityBot
If there's a "Disallow: /" under any of those, that model is locked out. If they aren't mentioned at all, you're open to them by default, which is fine.

That's the whole check for the part that's genuinely black and white.

Now the honest caveat, because I'd rather you trust the rest of this. Getting crawled is necessary, not sufficient. It lets you be cited. It doesn't make you cited.

We ran our on-site "readiness" score against real citations across 44 domains. The correlation was basically nothing (Pearson around -0.08). What predicted citations wasn't markup or content depth, it was brand prominence. Well-known domains got cited, everyone else mostly got zero, no matter how clean their SEO was. Small sample, one model, one snapshot, so don't treat it as a study. But it made me skeptical of every tool that promises "optimize on-page, get cited" (including the pull to oversell my own). The crawl gate is real. The rest is mostly being someone the model already trusts.

So the practical takeaway is small and boring: make sure you aren't accidentally locked out. That's the one move with a guaranteed effect. Everything after it is a slower game.

If you'd rather not read robots.txt by hand across a pile of sites, the CLI is open source (causabi-geo on PyPI) and there's a hosted version at causabi.com. Either way, happy to answer questions in the comments.

We Checked Whether On-Site SEO Predicts AI Citations. The Data Says Mostly No.

Mikhail Shadrin — Wed, 15 Jul 2026 21:07:09 +0000

Every GEO ("generative engine optimization") tool, including ours until
recently, sells some version of the same pitch: fix your robots.txt, add
Schema.org markup, write FAQ schema, and AI engines will cite you more.

We build one of these tools — Causabi scans sites for AI-crawler readiness
and generates fix files (robots.txt, llms.txt, JSON-LD, FAQ blocks). As part
of validating our own scoring weights, we ran the numbers on whether the
score actually predicts getting cited. Short version: it mostly doesn't,
once brand prominence is in the picture.

What we measured

We scored 44 domains on a 6-category on-site readiness algorithm:

robots.txt (AI bots allowed or blocked)
Schema.org (Organization/LocalBusiness JSON-LD completeness)
FAQ schema (FAQPage markup, 3+ entries)
content depth/structure
brand/NAP signals
freshness (dateModified, recency)

Then we checked how often each domain actually got cited by an AI engine
(Claude, via its web-search tool, one measurement window, a fixed prompt set
per domain).

What we found

We ran the check twice, and I'll give you both runs because the difference
between them is itself informative:

Run 1 (July 2): Claude only, n=44 usable domains. Score vs. citation rate: Pearson r = -0.078, Spearman ρ = -0.028. 86% of domains got zero citations regardless of score.
Run 2 (July 12): Claude + Gemini, n=41 usable. Spearman ρ = +0.084 (p = 0.60), Pearson r = +0.148 (p = 0.35). Zero-citation share dropped to 58.5% — Gemini names domains and brands much more freely than Claude.

Note the sign flipped between runs (-0.03 → +0.08). At these sample sizes
and p-values that's noise, and that's exactly the point: there is no
statistically significant relationship between on-site readiness score and
citation rate in either direction. If the correlation were real and strong,
two runs ten days apart wouldn't disagree on the sign.

The domains that did get cited clustered almost entirely by brand prominence — well-known domains got cited at a noticeably higher rate (~0.16 of prompts in run 1) than everyone else (~0 for the rest of the sample), regardless of how well-optimized their markup was.

Why I'm not overselling this

n=41-44 is small. This is an internal validation exercise for our own
product, not a peer-reviewed study, and I don't want it read as one.
"No significant correlation found" is not the same claim as "we proved
there is no relationship" — at this sample size we can't prove a negative.
Specific caveats:

Two engines so far (Claude, Gemini). Citation behavior differs meaningfully across ChatGPT, Grok, and Perplexity — we haven't run the same check across all of them yet.
One time window, no longitudinal before/after. We didn't take a domain, improve its score, and watch citations change over months. That's the actually convincing experiment and we haven't run it yet.
Prompt-domain matching wasn't blind. Some prompts were picked because a domain plausibly related to that topic, which likely biases toward domains that would get mentioned anyway.
"Brand prominence" is a fuzzy variable that probably absorbs some real content-quality signal we're not capturing separately. We can't fully rule out that what looks like "brand wins" is partly "genuinely better/more authoritative content wins," which on-site markup scoring doesn't measure.

What we still think is true, with more confidence

Some things aren't correlational guesses — they're closer to mechanical
facts:

robots.txt blocking is binary. If GPTBot, ClaudeBot, or similar are disallowed, that engine cites you zero times, by construction. About 89% of sites we've scanned block at least one AI crawler by default, usually by accident (a blanket Disallow: / that predates AI bots existing).
FAQ schema changes extraction, not inclusion. For content that's already in an engine's consideration set, structuring it as self-contained Q&A chunks seems to affect whether it gets pulled into a RAG-style citation — this lines up with published research on chunking behavior. But that's a "how you're cited" lever, not a "whether you're cited" lever.

Where that leaves the product

We're rewriting our own copy to say what the score actually measures:
AI-crawler readiness and machine-readability, not citation probability. No
tool — ours included — can promise the second one. If your on-site work is
mostly aimed at "getting cited more," the more binding constraint for most
sites is probably brand/mentions elsewhere, not another Schema.org type.

The scoring engine and fix generator are open source (MIT) if you want to
see the logic or run it on your own site without touching our SaaS:

pip install causabi-geo
geo-optimizer analyze https://yourdomain.com
geo-optimizer fix https://yourdomain.com --output ./out

Repo: https://github.com/SHADRINMMM/causabi-geo
Site (hosted version + monitoring): https://causabi.com