Is Your Site Even Visible to ChatGPT?

You've optimized for Google. Sitemap's clean, Core Web Vitals are green, structured data validates. But when someone asks ChatGPT or Perplexity about your product, you're nowhere in the answer — and a competitor with a worse site is.

The likely reason: the crawlers that feed AI answers aren't Googlebot, and they don't see your site the way Googlebot does.

Meet the other crawlers

• GPTBot (OpenAI) — crawls for model training and, via a separate retrieval path, for real-time browsing/citation in ChatGPT answers.
• ClaudeBot (Anthropic) — same idea, powers Claude's web search and citations.
• PerplexityBot — crawls specifically to answer live queries with citations; this is the one most directly tied to "will I get mentioned."
• Google-Extended — a separate opt-in signal from regular Googlebot, controls whether Google's AI features can use your content.

Each respects robots.txt, but as a distinct user-agent — blocking Googlebot doesn't block these, and blocking one doesn't block the others. Most sites have never explicitly considered any of them.

Why a site that ranks fine can still be invisible

1. robots.txt blocks it, often by accident. Some site generators and "AI protection" plugins add blanket Disallow rules for GPTBot/CCBot/etc. without the owner realizing it also kills citation eligibility, not just training-data scraping. Check yours:

curl -s https://yoursite.com/robots.txt | grep -A2 -i "gptbot\|claudebot\|perplexitybot"

2. The content is client-rendered only. This is the big one. Traditional SEO crawlers (Googlebot) execute JavaScript and wait for your SPA to render. Several AI crawlers fetch raw HTML and move on — no JS execution. If your content only exists after a React/Vue mount, an AI crawler may see an empty <div id="root"> and nothing else.

Test what a crawler actually receives, vs. what your browser renders:

curl -s -A "GPTBot" https://yoursite.com/ | grep -o '<title>.*</title>'

If that title tag (or the body content you care about) is missing or generic, but your browser shows it fine — that's your JS-rendering gap.

3. No structured data, or broken structured data. AI answer engines lean on schema.org JSON-LD to disambiguate what a page is actually about — product, article, FAQ, organization — rather than inferring it purely from prose. A page with solid content but zero structured data is harder for a retrieval system to confidently cite.

4. No llms.txt. An emerging, not-yet-universal convention — a plain-text file at /llms.txt giving a curated, LLM-readable summary of what your site is and where the important pages are, the same spirit as sitemap.xml but written for language models instead of search indexers. Not every crawler uses it yet, but it costs almost nothing to add and directly addresses the "can a model even summarize what we do" problem.

5. Stale noindex leftovers. Staging-environment meta tags (<meta name="robots" content="noindex">) that never got removed after launch block everything — traditional and AI crawlers alike.

A five-minute self-check

1. curl -A "GPTBot" yoursite.com/robots.txt — confirm you're not accidentally blocked.
2. curl -A "GPTBot" yoursite.com/ and diff it against what your browser renders — confirm the content is actually in the raw HTML.
3. Validate your structured data (Google's Rich Results Test works fine for this even though it's Google-branded — it's just a schema.org validator).
4. Check if /llms.txt exists. If not, it's a 20-minute add.
5. Grep your codebase for leftover noindex tags.

Automating this

We ran into this exact problem building RankCLI — the open MCP server (npx @rankcli/mcp-server, no signup) runs GEO checks for all of the above alongside the usual technical-SEO audit, so it's one pass instead of five manual curl commands every time you ship. Free tier is genuinely free; the hosted layer just adds scheduling and auto-fix PRs on top.

But even without any tool, the five checks above take less time than writing this sentence took, and most sites have never run them once.