Is Your Site Even Visible to ChatGPT? Auditing "GEO" (AI Crawler Access)

Brian Sam-Bodden — Sun, 06 Sep 2026 17:08:43 +0000

If you ship a robots.txt once and forget about it, there's a decent chance you're invisible to the crawlers that actually feed answers into ChatGPT, Perplexity, and Claude right now — and most SEO tooling won't tell you, because it was built for Googlebot, not GPTBot.

I spent a chunk of this year building an SEO CLI (more on that at the end), and the AI-crawler side turned out to be the part with the least existing tooling. So here's what I learned auditing it, plus a few checks you can run on your own site in the next five minutes.

The crawlers that matter now

Google isn't the only bot reading your site anymore. If you care about showing up in AI-generated answers, these are the user agents worth knowing:

Crawler	Company	Purpose
`GPTBot`	OpenAI	Training data + (separately) `ChatGPT-User` for live browsing
`ClaudeBot`	Anthropic	Training + retrieval
`PerplexityBot`	Perplexity	Powers live citations in Perplexity answers
`Google-Extended`	Google	Opts you in/out of Gemini/AI Overviews specifically — separate from regular Googlebot
`Bingbot`	Microsoft	Feeds Copilot as well as classic Bing search

The catch: a lot of robots.txt files either predate these bots entirely, or block them "for safety" without realizing that also means zero chance of being cited when someone asks an AI assistant a question your content actually answers.

Check #1: What does your robots.txt actually say?

curl -s https://yoursite.com/robots.txt

Look specifically for blanket rules like:

User-agent: *
Disallow: /

That blocks everything, including AI crawlers, by default. More common and sneakier: an explicit block on just the AI bots because someone copy-pasted a "protect your content from AI" snippet without thinking through the tradeoff:

User-agent: GPTBot
Disallow: /

If your business model depends on being found, this is usually the wrong default. You can always carve out exceptions for specific paths instead of an all-or-nothing block.

Check #2: Can the crawler actually render your content?

This is the one people miss. Most AI crawlers do not execute JavaScript. If your page is a client-rendered React/Vue app that fetches content after load, GPTBot sees an empty <div id="root"> and nothing else — even if your robots.txt is wide open.

Quick test:

curl -s -A "GPTBot" https://yoursite.com | grep -o "<title>.*</title>"

If that comes back empty or with a generic loading-state title, that's your answer. Server-side rendering, static generation, or at minimum pre-rendering for bot user agents will fix it.

Check #3: Is your content structured for extraction?

Even with access, LLM-facing crawlers do better with content that's structured to be quoted, not just read:

Clear H2/H3 hierarchy instead of one wall-of-text <div>
FAQ sections with actual question-formatted headings (these get lifted almost verbatim into AI answers)
JSON-LD structured data (Article, FAQPage, HowTo) — LLM crawlers use this as a strong trust/extraction signal, same as it always helped rich results
Direct, extractable answers near the top of a section rather than buried after three paragraphs of preamble

Check #4: Citation readiness

If a model did want to cite you, would it have what it needs? Author attribution, a clear publish/update date, and a canonical URL all factor into whether a crawler treats your page as a citable source versus content to skim and move past.

Putting it together

I ended up building this into a CLI tool (RankCLI) because I wanted these checks to live in CI, not in a dashboard I'd forget to open. There's also a free MCP server if you want to run these checks straight from Claude, Cursor, or any other MCP-compatible AI host — no signup, nothing sent to a server, it just reads the HTML your MCP host already fetched:

npx @rankcli/mcp-server

or point an MCP client at the hosted version:

{
  "mcpServers": {
    "rankcli": { "url": "https://mcp.rankcli.dev/mcp/free" }
  }
}

But even without any tool, the four checks above take about five minutes and will tell you more about your AI-search visibility than most SEO audits currently do. Worth doing before you spend more time optimizing for a Google result page that's increasingly not where people are looking.

What are you seeing on your own sites — anyone actually blocking GPTBot on purpose, or is it mostly accidental?