<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Xiaoli</title>
    <description>The latest articles on DEV Community by Xiaoli (@neocraft).</description>
    <link>https://dev.to/neocraft</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4248704%2F000d5021-e1c2-4b86-b788-3166877c4a67.png</url>
      <title>DEV Community: Xiaoli</title>
      <link>https://dev.to/neocraft</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/neocraft"/>
    <language>en</language>
    <item>
      <title>Is your website visible to ChatGPT and Perplexity? A practical GEO checklist (and how to automate it)</title>
      <dc:creator>Xiaoli</dc:creator>
      <pubDate>Tue, 17 Nov 2026 02:34:20 +0000</pubDate>
      <link>https://dev.to/neocraft/is-your-website-visible-to-chatgpt-and-perplexity-a-practical-geo-checklist-and-how-to-automate-3773</link>
      <guid>https://dev.to/neocraft/is-your-website-visible-to-chatgpt-and-perplexity-a-practical-geo-checklist-and-how-to-automate-3773</guid>
      <description>&lt;p&gt;When someone asks ChatGPT, Claude or Perplexity "what's the best tool for X?", the answer cites a handful of pages. If an AI crawler can't fetch your page, or fetches it and finds an empty JavaScript shell, you won't be one of them, however good your content is.&lt;/p&gt;

&lt;p&gt;"GEO" (generative engine optimization) gets sold as a new kind of magic. Most of it is plain technical hygiene that you can check in an afternoon. Below is the checklist I use, with commands you can run and a short script that automates the boring parts. Every claim about how a crawler behaves links to the vendor's own documentation.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Know which bots you're dealing with
&lt;/h2&gt;

&lt;p&gt;Each AI company runs several user agents, and they do different jobs. Blocking the "training" bot does &lt;strong&gt;not&lt;/strong&gt; have to remove you from AI search results, and the reverse is also true.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Company&lt;/th&gt;
&lt;th&gt;Token&lt;/th&gt;
&lt;th&gt;What it's for (per vendor docs)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;OpenAI&lt;/td&gt;
&lt;td&gt;&lt;code&gt;GPTBot&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Content that may be used to train foundation models&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenAI&lt;/td&gt;
&lt;td&gt;&lt;code&gt;OAI-SearchBot&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Surfacing sites in ChatGPT search results&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenAI&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ChatGPT-User&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Visits triggered by a user's request. OpenAI says robots.txt rules &lt;em&gt;may not apply&lt;/em&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Anthropic&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ClaudeBot&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Content that may be used for model training&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Anthropic&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Claude-SearchBot&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Improving search results quality&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Anthropic&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Claude-User&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Fetches when a Claude user asks a question&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Perplexity&lt;/td&gt;
&lt;td&gt;&lt;code&gt;PerplexityBot&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Surfacing and linking sites in Perplexity search (Perplexity says it is not used for training foundation models)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Perplexity&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Perplexity-User&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;User-requested fetches. Perplexity says it &lt;em&gt;generally ignores&lt;/em&gt; robots.txt&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Google&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Google-Extended&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;A robots.txt token, not a separate crawler. It controls Gemini training and grounding, and Google says it does not affect inclusion or ranking in Google Search&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Google&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Googlebot&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Google Search, &lt;em&gt;including&lt;/em&gt; AI Overviews and AI Mode&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The last row matters: you can't opt out of AI Overviews with &lt;code&gt;Google-Extended&lt;/code&gt;. For Google, the AI features are part of Search.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. robots.txt: decide on training and search separately
&lt;/h2&gt;

&lt;p&gt;A sensible default for many businesses is "don't train on my content, but do cite me in answers". In robots.txt:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# AI search bots: allowed (user-triggered fetchers fall under * below)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Model training: not allowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mistakes I see often:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A blanket &lt;code&gt;User-agent: *&lt;/code&gt; + &lt;code&gt;Disallow: /&lt;/code&gt;&lt;/strong&gt; left over from a staging site.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"Block all AI" lists copied from a blog post&lt;/strong&gt;, which also block the search bots, so the site vanishes from AI answers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;robots.txt answering with a server error (5xx).&lt;/strong&gt; Google, for one, limits or pauses crawling while it can't read the file, whereas a 4xx (except 429) is treated as "no restrictions" (&lt;a href="https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec" rel="noopener noreferrer"&gt;Google's robots.txt spec&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Expecting instant changes.&lt;/strong&gt; OpenAI says it can take about 24 hours for its search systems to pick up a robots.txt change.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Check yours: &lt;code&gt;curl -s https://www.example.com/robots.txt&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Check your CDN / firewall, not just robots.txt
&lt;/h2&gt;

&lt;p&gt;robots.txt is a polite request. Your CDN or WAF can block AI crawlers &lt;em&gt;before&lt;/em&gt; they ever read it, and then your carefully written rules don't matter. On Cloudflare the setting lives under &lt;strong&gt;Security &amp;gt; Settings &amp;gt; AI Crawler and Agent access policies&lt;/strong&gt;, with separate actions for Search, Agent and Training traffic. Other CDNs and security plugins have similar toggles. Perplexity even publishes WAF allow-list guidance for its bots.&lt;/p&gt;

&lt;p&gt;You can get a rough idea with a request using a bot's user agent:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -s -o /dev/null -w &lt;span class="s2"&gt;"%{http_code}\n"&lt;/span&gt; \
  -A &lt;span class="s2"&gt;"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"&lt;/span&gt; \
  https://www.example.com/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Treat the result as a hint, not proof. Many firewalls verify real bots by IP range (the vendors publish them), so a spoofed user agent can be handled differently from the real crawler. The reliable signals are your firewall's settings and its logs.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Make sure the content is in the HTML
&lt;/h2&gt;

&lt;p&gt;This is the one that hurts single-page apps the most. A Vercel and MERJ analysis of real crawler traffic (December 2024) found that none of the major AI crawlers they observed rendered JavaScript. That included OpenAI's bots, ClaudeBot and PerplexityBot. They downloaded JS files but didn't execute them. Google's Gemini, which uses Googlebot's infrastructure, was the exception. Crawlers change, but you shouldn't count on JavaScript being executed.&lt;/p&gt;

&lt;p&gt;Test it the way a crawler sees it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -s https://www.example.com/pricing &lt;span class="o"&gt;|&lt;/span&gt; grep -i &lt;span class="s2"&gt;"per month"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If your prices, product descriptions or answers only appear after client-side rendering, use server-side rendering or static generation (Next.js, Nuxt, Astro, SvelteKit and similar all support it), or at least pre-render the pages that matter.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Don't hide your text by accident
&lt;/h2&gt;

&lt;p&gt;Snippet controls are legitimate, but they're easy to set by mistake through a theme, plugin or CDN header:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;&amp;lt;meta name="robots" content="nosnippet"&amp;gt;&lt;/code&gt; or &lt;code&gt;max-snippet:0&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;An &lt;code&gt;X-Robots-Tag: nosnippet&lt;/code&gt; HTTP header&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;data-nosnippet&lt;/code&gt; on the main content block&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Google's documentation says &lt;code&gt;nosnippet&lt;/code&gt; applies to AI Overviews and AI Mode too, and also stops the content being used as a direct input for them. Check with &lt;code&gt;curl -sI https://www.example.com/ | grep -i x-robots-tag&lt;/code&gt; and view the page source.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Structured data that matches the page
&lt;/h2&gt;

&lt;p&gt;Structured data won't guarantee that an LLM cites you. Nobody has public proof of that. What it does is make your key facts machine-readable and unambiguous: who you are, what you sell, who wrote an article and when. The essentials:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;Organization&lt;/code&gt; (or &lt;code&gt;LocalBusiness&lt;/code&gt;) with &lt;code&gt;name&lt;/code&gt;, &lt;code&gt;logo&lt;/code&gt;, &lt;code&gt;url&lt;/code&gt; and &lt;code&gt;sameAs&lt;/code&gt; links to your official profiles&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Article&lt;/code&gt; / &lt;code&gt;BlogPosting&lt;/code&gt; with &lt;code&gt;author&lt;/code&gt;, &lt;code&gt;datePublished&lt;/code&gt; and &lt;code&gt;dateModified&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Product&lt;/code&gt; with an &lt;code&gt;Offer&lt;/code&gt; (&lt;code&gt;price&lt;/code&gt;, &lt;code&gt;priceCurrency&lt;/code&gt;, &lt;code&gt;availability&lt;/code&gt;) on product pages
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight html"&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;script&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"application/ld+json"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="s2"&gt;"@context"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://schema.org"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;"@type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Organization"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Example Ltd"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://www.example.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;"logo"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://www.example.com/logo.png"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;"sameAs"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"https://www.linkedin.com/company/example"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"https://github.com/example"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;/script&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Follow Google's rule: don't mark up content that readers can't see on the page.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Write pages that can be quoted
&lt;/h2&gt;

&lt;p&gt;Google says there are "no additional requirements" to appear in AI Overviews or AI Mode beyond normal SEO best practices. In practice, pages that are easy to quote tend to share a few traits:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Headings that match real questions ("How much does X cost?"), with the direct answer in the first sentence below them&lt;/li&gt;
&lt;li&gt;Lists and tables for comparisons, specs and steps&lt;/li&gt;
&lt;li&gt;A visible author, a visible "last updated" date, and easy-to-find About and Contact pages&lt;/li&gt;
&lt;li&gt;Links to your sources when you state facts or numbers&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  8. Freshness signals that are actually true
&lt;/h2&gt;

&lt;p&gt;Keep an XML sitemap with accurate &lt;code&gt;&amp;lt;lastmod&amp;gt;&lt;/code&gt; dates. Google says it uses &lt;code&gt;lastmod&lt;/code&gt; only if it's "consistently and verifiably accurate", so a CMS that stamps today's date on every page teaches crawlers to ignore it.&lt;/p&gt;

&lt;h2&gt;
  
  
  9. llms.txt: cheap, optional, and not a ranking factor
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://llmstxt.org/" rel="noopener noreferrer"&gt;llms.txt&lt;/a&gt; is a proposal by Jeremy Howard (first published in September 2024, v2 in 2026). It's a Markdown file at &lt;code&gt;/llms.txt&lt;/code&gt; that gives agents a short overview of your site and links to clean Markdown versions of key pages. Coding agents use it most, on developer documentation, and OpenAI, Anthropic and Google publish one for their own API docs.&lt;/p&gt;

&lt;p&gt;To keep expectations realistic: I couldn't find any statement from an AI search provider that llms.txt affects whether you get cited, and Google's guidance says no special optimization is needed. It takes ten minutes, so add one if your site has docs or a product that agents need to understand. Just don't expect it to fix problems from steps 2 to 4.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;&lt;span class="gu"&gt;# Example Ltd
&lt;/span&gt;
&lt;span class="ge"&gt;&amp;gt; Example Ltd makes invoicing software for freelancers in the EU.
&lt;/span&gt;
&lt;span class="gu"&gt;## Docs
&lt;/span&gt;&lt;span class="k"&gt;- &lt;/span&gt;[Pricing](https://www.example.com/pricing.md): plans and limits
&lt;span class="k"&gt;- &lt;/span&gt;[API reference](https://www.example.com/docs/api.md)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Automate the basics (Python, no dependencies)
&lt;/h2&gt;

&lt;p&gt;This script checks robots.txt for the bots above, looks for llms.txt, and checks that a key sentence appears in the raw HTML with no JavaScript:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="s2"&gt;"""Quick AI-visibility check: robots.txt rules for AI crawlers, llms.txt, server-rendered text.
Usage: python ai_visibility_check.py https://www.example.com https://www.example.com/pricing "per month"
"""&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;robotparser&lt;/span&gt;

&lt;span class="no"&gt;BOTS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"GPTBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"OAI-SearchBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ChatGPT-User"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ClaudeBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Claude-SearchBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"Claude-User"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"PerplexityBot"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Perplexity-User"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Google-Extended"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"User-Agent"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"ai-visibility-check/1.0"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;15&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"utf-8"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"replace"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;Exception&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;getattr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"code"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;

&lt;span class="n"&gt;site&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;phrase&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;rp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;robotparser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;RobotFileParser&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;site&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;"/robots.txt"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;rp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;splitlines&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"robots.txt: HTTP {status}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;bot&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="no"&gt;BOTS&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"  {bot:17} {'allowed' if rp.can_fetch(bot, page) else 'BLOCKED'}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;site&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;"/llms.txt"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"llms.txt: {'present' if status == 200 else f'not found (HTTP {status})'}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"'{phrase}' in raw HTML (no JavaScript): {'yes' if phrase.lower() in html.lower() else 'NO'}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Python's &lt;code&gt;robotparser&lt;/code&gt; is a simplified implementation: it applies rules in file order rather than Google's "most specific rule wins". Use it as a smoke test, and confirm edge cases with the vendor's own tools. Run it in CI or on a weekly cron so that a theme update or a new CDN rule can't quietly make you invisible.&lt;/p&gt;

&lt;h2&gt;
  
  
  If you'd rather not maintain it yourself
&lt;/h2&gt;

&lt;p&gt;I also built a hosted version: &lt;a href="https://apify.com/neocraft/website-audit-all-in-one" rel="noopener noreferrer"&gt;Website Audit All-in-One&lt;/a&gt; on Apify. It runs the checks above (AI crawler rules, Content Signals, snippet controls, JSON-LD quality, rendered vs raw content, llms.txt, sitemap freshness) together with classic SEO, security headers and speed checks. It returns a score with prioritized fixes and can compare runs week to week. It costs $0.49 per site (10 pages), with no subscription. The checklist above works just as well without it.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Disclosure: I built this tool; this article was drafted by an AI assistant and reviewed by me.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Sources&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;OpenAI, &lt;a href="https://platform.openai.com/docs/bots" rel="noopener noreferrer"&gt;Overview of OpenAI crawlers&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Anthropic, &lt;a href="https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler" rel="noopener noreferrer"&gt;Does Anthropic crawl data from the web, and how can site owners block the crawler?&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Perplexity, &lt;a href="https://docs.perplexity.ai/guides/bots" rel="noopener noreferrer"&gt;Perplexity crawlers&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Google, &lt;a href="https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec" rel="noopener noreferrer"&gt;robots.txt specification&lt;/a&gt;, &lt;a href="https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers" rel="noopener noreferrer"&gt;Google's common crawlers (Google-Extended)&lt;/a&gt;, &lt;a href="https://developers.google.com/search/docs/appearance/ai-features" rel="noopener noreferrer"&gt;AI features and your website&lt;/a&gt;, &lt;a href="https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag" rel="noopener noreferrer"&gt;Robots meta tag, data-nosnippet and X-Robots-Tag&lt;/a&gt;, &lt;a href="https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap" rel="noopener noreferrer"&gt;Build a sitemap&lt;/a&gt;, &lt;a href="https://developers.google.com/search/docs/appearance/structured-data/sd-policies" rel="noopener noreferrer"&gt;Structured data general guidelines&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Vercel and MERJ, &lt;a href="https://vercel.com/blog/the-rise-of-the-ai-crawler" rel="noopener noreferrer"&gt;The rise of the AI crawler&lt;/a&gt; (December 2024)&lt;/li&gt;
&lt;li&gt;Cloudflare, &lt;a href="https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/" rel="noopener noreferrer"&gt;AI Crawler and Agent access policies&lt;/a&gt;, &lt;a href="https://contentsignals.org/" rel="noopener noreferrer"&gt;Content Signals&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Jeremy Howard, &lt;a href="https://llmstxt.org/" rel="noopener noreferrer"&gt;The /llms.txt file&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>seo</category>
      <category>ai</category>
      <category>webdev</category>
      <category>tutorial</category>
    </item>
  </channel>
</rss>
