DEV Community

Xiaoli
Xiaoli

Posted on

Is your website visible to ChatGPT and Perplexity? A practical GEO checklist (and how to automate it)

When someone asks ChatGPT, Claude or Perplexity "what's the best tool for X?", the answer cites a handful of pages. If an AI crawler can't fetch your page, or fetches it and finds an empty JavaScript shell, you won't be one of them, however good your content is.

"GEO" (generative engine optimization) gets sold as a new kind of magic. Most of it is plain technical hygiene that you can check in an afternoon. Below is the checklist I use, with commands you can run and a short script that automates the boring parts. Every claim about how a crawler behaves links to the vendor's own documentation.

1. Know which bots you're dealing with

Each AI company runs several user agents, and they do different jobs. Blocking the "training" bot does not have to remove you from AI search results, and the reverse is also true.

Company Token What it's for (per vendor docs)
OpenAI GPTBot Content that may be used to train foundation models
OpenAI OAI-SearchBot Surfacing sites in ChatGPT search results
OpenAI ChatGPT-User Visits triggered by a user's request. OpenAI says robots.txt rules may not apply
Anthropic ClaudeBot Content that may be used for model training
Anthropic Claude-SearchBot Improving search results quality
Anthropic Claude-User Fetches when a Claude user asks a question
Perplexity PerplexityBot Surfacing and linking sites in Perplexity search (Perplexity says it is not used for training foundation models)
Perplexity Perplexity-User User-requested fetches. Perplexity says it generally ignores robots.txt
Google Google-Extended A robots.txt token, not a separate crawler. It controls Gemini training and grounding, and Google says it does not affect inclusion or ranking in Google Search
Google Googlebot Google Search, including AI Overviews and AI Mode

The last row matters: you can't opt out of AI Overviews with Google-Extended. For Google, the AI features are part of Search.

2. robots.txt: decide on training and search separately

A sensible default for many businesses is "don't train on my content, but do cite me in answers". In robots.txt:

# AI search bots: allowed (user-triggered fetchers fall under * below)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Model training: not allowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml
Enter fullscreen mode Exit fullscreen mode

Mistakes I see often:

  • A blanket User-agent: * + Disallow: / left over from a staging site.
  • "Block all AI" lists copied from a blog post, which also block the search bots, so the site vanishes from AI answers.
  • robots.txt answering with a server error (5xx). Google, for one, limits or pauses crawling while it can't read the file, whereas a 4xx (except 429) is treated as "no restrictions" (Google's robots.txt spec).
  • Expecting instant changes. OpenAI says it can take about 24 hours for its search systems to pick up a robots.txt change.

Check yours: curl -s https://www.example.com/robots.txt.

3. Check your CDN / firewall, not just robots.txt

robots.txt is a polite request. Your CDN or WAF can block AI crawlers before they ever read it, and then your carefully written rules don't matter. On Cloudflare the setting lives under Security > Settings > AI Crawler and Agent access policies, with separate actions for Search, Agent and Training traffic. Other CDNs and security plugins have similar toggles. Perplexity even publishes WAF allow-list guidance for its bots.

You can get a rough idea with a request using a bot's user agent:

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  https://www.example.com/
Enter fullscreen mode Exit fullscreen mode

Treat the result as a hint, not proof. Many firewalls verify real bots by IP range (the vendors publish them), so a spoofed user agent can be handled differently from the real crawler. The reliable signals are your firewall's settings and its logs.

4. Make sure the content is in the HTML

This is the one that hurts single-page apps the most. A Vercel and MERJ analysis of real crawler traffic (December 2024) found that none of the major AI crawlers they observed rendered JavaScript. That included OpenAI's bots, ClaudeBot and PerplexityBot. They downloaded JS files but didn't execute them. Google's Gemini, which uses Googlebot's infrastructure, was the exception. Crawlers change, but you shouldn't count on JavaScript being executed.

Test it the way a crawler sees it:

curl -s https://www.example.com/pricing | grep -i "per month"
Enter fullscreen mode Exit fullscreen mode

If your prices, product descriptions or answers only appear after client-side rendering, use server-side rendering or static generation (Next.js, Nuxt, Astro, SvelteKit and similar all support it), or at least pre-render the pages that matter.

5. Don't hide your text by accident

Snippet controls are legitimate, but they're easy to set by mistake through a theme, plugin or CDN header:

  • <meta name="robots" content="nosnippet"> or max-snippet:0
  • An X-Robots-Tag: nosnippet HTTP header
  • data-nosnippet on the main content block

Google's documentation says nosnippet applies to AI Overviews and AI Mode too, and also stops the content being used as a direct input for them. Check with curl -sI https://www.example.com/ | grep -i x-robots-tag and view the page source.

6. Structured data that matches the page

Structured data won't guarantee that an LLM cites you. Nobody has public proof of that. What it does is make your key facts machine-readable and unambiguous: who you are, what you sell, who wrote an article and when. The essentials:

  • Organization (or LocalBusiness) with name, logo, url and sameAs links to your official profiles
  • Article / BlogPosting with author, datePublished and dateModified
  • Product with an Offer (price, priceCurrency, availability) on product pages
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Example Ltd",
  "url": "https://www.example.com",
  "logo": "https://www.example.com/logo.png",
  "sameAs": ["https://www.linkedin.com/company/example", "https://github.com/example"]
}
</script>
Enter fullscreen mode Exit fullscreen mode

Follow Google's rule: don't mark up content that readers can't see on the page.

7. Write pages that can be quoted

Google says there are "no additional requirements" to appear in AI Overviews or AI Mode beyond normal SEO best practices. In practice, pages that are easy to quote tend to share a few traits:

  • Headings that match real questions ("How much does X cost?"), with the direct answer in the first sentence below them
  • Lists and tables for comparisons, specs and steps
  • A visible author, a visible "last updated" date, and easy-to-find About and Contact pages
  • Links to your sources when you state facts or numbers

8. Freshness signals that are actually true

Keep an XML sitemap with accurate <lastmod> dates. Google says it uses lastmod only if it's "consistently and verifiably accurate", so a CMS that stamps today's date on every page teaches crawlers to ignore it.

9. llms.txt: cheap, optional, and not a ranking factor

llms.txt is a proposal by Jeremy Howard (first published in September 2024, v2 in 2026). It's a Markdown file at /llms.txt that gives agents a short overview of your site and links to clean Markdown versions of key pages. Coding agents use it most, on developer documentation, and OpenAI, Anthropic and Google publish one for their own API docs.

To keep expectations realistic: I couldn't find any statement from an AI search provider that llms.txt affects whether you get cited, and Google's guidance says no special optimization is needed. It takes ten minutes, so add one if your site has docs or a product that agents need to understand. Just don't expect it to fix problems from steps 2 to 4.

# Example Ltd

> Example Ltd makes invoicing software for freelancers in the EU.

## Docs
- [Pricing](https://www.example.com/pricing.md): plans and limits
- [API reference](https://www.example.com/docs/api.md)
Enter fullscreen mode Exit fullscreen mode

Automate the basics (Python, no dependencies)

This script checks robots.txt for the bots above, looks for llms.txt, and checks that a key sentence appears in the raw HTML with no JavaScript:

"""Quick AI-visibility check: robots.txt rules for AI crawlers, llms.txt, server-rendered text.
Usage: python ai_visibility_check.py https://www.example.com https://www.example.com/pricing "per month"
"""
import sys
import urllib.request
import urllib.robotparser

BOTS = ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "Claude-SearchBot",
        "Claude-User", "PerplexityBot", "Perplexity-User", "Google-Extended"]

def fetch(url):
    req = urllib.request.Request(url, headers={"User-Agent": "ai-visibility-check/1.0"})
    try:
        with urllib.request.urlopen(req, timeout=15) as r:
            return r.status, r.read().decode("utf-8", "replace")
    except Exception as e:
        return getattr(e, "code", None), ""

site, page, phrase = sys.argv[1].rstrip("/"), sys.argv[2], sys.argv[3]

rp = urllib.robotparser.RobotFileParser()
status, body = fetch(site + "/robots.txt")
rp.parse(body.splitlines())
print(f"robots.txt: HTTP {status}")
for bot in BOTS:
    print(f"  {bot:17} {'allowed' if rp.can_fetch(bot, page) else 'BLOCKED'}")

status, _ = fetch(site + "/llms.txt")
print(f"llms.txt: {'present' if status == 200 else f'not found (HTTP {status})'}")

status, html = fetch(page)
print(f"'{phrase}' in raw HTML (no JavaScript): {'yes' if phrase.lower() in html.lower() else 'NO'}")
Enter fullscreen mode Exit fullscreen mode

Python's robotparser is a simplified implementation: it applies rules in file order rather than Google's "most specific rule wins". Use it as a smoke test, and confirm edge cases with the vendor's own tools. Run it in CI or on a weekly cron so that a theme update or a new CDN rule can't quietly make you invisible.

If you'd rather not maintain it yourself

I also built a hosted version: Website Audit All-in-One on Apify. It runs the checks above (AI crawler rules, Content Signals, snippet controls, JSON-LD quality, rendered vs raw content, llms.txt, sitemap freshness) together with classic SEO, security headers and speed checks. It returns a score with prioritized fixes and can compare runs week to week. It costs $0.49 per site (10 pages), with no subscription. The checklist above works just as well without it.


Disclosure: I built this tool; this article was drafted by an AI assistant and reviewed by me.

Sources

Top comments (0)