<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: גיא עמוס</title>
    <description>The latest articles on DEV Community by גיא עמוס (@__bcc270a3023440da).</description>
    <link>https://dev.to/__bcc270a3023440da</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4257019%2F82640a0a-8161-42b5-9422-5633f16ddcd8.jpg</url>
      <title>DEV Community: גיא עמוס</title>
      <link>https://dev.to/__bcc270a3023440da</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/__bcc270a3023440da"/>
    <language>en</language>
    <item>
      <title>Why Your Website Is Invisible to ChatGPT &amp; Perplexity — And How to Audit It in 10 Seconds</title>
      <dc:creator>גיא עמוס</dc:creator>
      <pubDate>Sat, 21 Nov 2026 09:39:34 +0000</pubDate>
      <link>https://dev.to/__bcc270a3023440da/why-your-website-is-invisible-to-chatgpt-perplexity-and-how-to-audit-it-in-10-seconds-1ca8</link>
      <guid>https://dev.to/__bcc270a3023440da/why-your-website-is-invisible-to-chatgpt-perplexity-and-how-to-audit-it-in-10-seconds-1ca8</guid>
      <description>&lt;p&gt;More developers and buyers are discovering tools through ChatGPT Search, Perplexity, and Claude instead of clicking through ten blue links.&lt;/p&gt;

&lt;p&gt;If you search for advice on "Generative Engine Optimization (GEO)," most articles immediately pitch vague content tips or $99/month prompt-tracking dashboards. Before worrying about prompt rankings, though, there is a much simpler question to answer:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can AI crawlers even read your website's HTML?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;After auditing dozens of developer tools, SaaS landing pages, and documentation sites, I kept finding the same four technical blockers. Here is how each one breaks AI visibility — and how to test your site in 10 seconds.&lt;/p&gt;




&lt;h3&gt;
  
  
  1. Blocking Live AI Search Bots in &lt;code&gt;robots.txt&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;A lot of teams copied a blanket "block AI bots" snippet into &lt;code&gt;robots.txt&lt;/code&gt; back in 2023 to opt out of model training. The problem is that AI platforms now separate &lt;strong&gt;training crawlers&lt;/strong&gt; from &lt;strong&gt;live search &amp;amp; user-triggered fetchers&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;If you want to opt out of training while still appearing as a live cited source in ChatGPT Search, Perplexity, or Claude, your &lt;code&gt;robots.txt&lt;/code&gt; needs to distinguish them:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# Opt out of model training crawlers (optional)
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Allow live search and citation fetchers
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;Check your &lt;code&gt;robots.txt&lt;/code&gt; right now — you might be blocking &lt;code&gt;OAI-SearchBot&lt;/code&gt; or &lt;code&gt;PerplexityBot&lt;/code&gt; without realizing it.&lt;/p&gt;




&lt;h3&gt;
  
  
  2. The Sneaky &lt;code&gt;/llms.txt&lt;/code&gt; SPA 404 Trap
&lt;/h3&gt;

&lt;p&gt;The &lt;a href="https://llmstxt.org/" rel="noopener noreferrer"&gt;llms.txt specification&lt;/a&gt; gives LLMs a clean, Markdown index of your site at &lt;code&gt;/llms.txt&lt;/code&gt; (and optionally a full concatenated digest at &lt;code&gt;/llms-full.txt&lt;/code&gt;).&lt;/p&gt;

&lt;p&gt;Here is a bug that catches almost every Single-Page App (React, Vue, Vite, Cloudflare Pages, Netlify). Run this in your terminal against a typical SPA that doesn't have an &lt;code&gt;llms.txt&lt;/code&gt; file yet:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;curl -i https://example-spa.com/llms.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;Instead of &lt;code&gt;404 Not Found&lt;/code&gt;, the catch-all SPA router returns:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;HTTP/2 200
content-type: text/html; charset=utf-8

&amp;lt;!doctype html&amp;gt;
&amp;lt;html&amp;gt;&amp;lt;body&amp;gt;&amp;lt;div id="root"&amp;gt;&amp;lt;/div&amp;gt;&amp;lt;/body&amp;gt;&amp;lt;/html&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;Naive uptime or SEO checkers only check &lt;code&gt;status === 200&lt;/code&gt; and report "Valid &lt;code&gt;llms.txt&lt;/code&gt; detected." An AI crawler, meanwhile, receives an HTML page instead of Markdown.&lt;/p&gt;

&lt;p&gt;A real &lt;code&gt;/llms.txt&lt;/code&gt; should return &lt;code&gt;text/plain&lt;/code&gt; or &lt;code&gt;text/markdown&lt;/code&gt; and follow a clean heading-and-link structure:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# Acme API

&amp;gt; Acme API provides deterministic PDF invoice generation for Node.js and Python backends.

## Core Documentation
- [Quickstart](https://acme.dev/docs/quickstart): Authenticate and generate your first PDF in 3 minutes.
- [API Reference](https://acme.dev/docs/reference): Endpoints, request schemas, and webhook events.
- [Pricing](https://acme.dev/pricing): Usage tiers and rate limits.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;




&lt;h3&gt;
  
  
  3. Empty Raw HTML (The JS-Only SPA Trap)
&lt;/h3&gt;

&lt;p&gt;Googlebot runs a massive headless Chromium rendering pipeline. Most AI search fetchers do not — they optimize for fast, cheap HTTP fetches and parse the raw HTML response directly.&lt;/p&gt;

&lt;p&gt;Test what a non-JS fetcher actually sees on your homepage:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;curl -s https://yoursite.com | grep -o "&amp;lt;p&amp;gt;.*&amp;lt;/p&amp;gt;" | head -n 5
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;If your raw HTML is mostly &lt;code&gt;&amp;lt;script&amp;gt;&lt;/code&gt; bundles and an empty &lt;code&gt;&amp;lt;div id="root"&amp;gt;&amp;lt;/div&amp;gt;&lt;/code&gt;, AI crawlers see almost zero product copy, pricing, or documentation text. Server-side rendering (SSR), static site generation (SSG), or pre-rendering critical marketing and docs routes fixes this completely.&lt;/p&gt;




&lt;h3&gt;
  
  
  4. Missing or Malformed &lt;code&gt;JSON-LD&lt;/code&gt; &amp;amp; Citability Signals
&lt;/h3&gt;

&lt;p&gt;Structured data isn't just for Google rich snippets. Clean &lt;code&gt;JSON-LD&lt;/code&gt; (&lt;code&gt;Organization&lt;/code&gt;, &lt;code&gt;SoftwareApplication&lt;/code&gt;, &lt;code&gt;Article&lt;/code&gt;, &lt;code&gt;FAQPage&lt;/code&gt;) gives extraction pipelines unambiguous facts about what your tool does, its price, its documentation URL, and who maintains it.&lt;/p&gt;

&lt;p&gt;Common mistakes I see in the wild:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Trailing commas or unescaped quotes breaking &lt;code&gt;JSON.parse()&lt;/code&gt; inside &lt;code&gt;&amp;lt;script type="application/ld+json"&amp;gt;&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Missing canonical URLs causing duplicate docs versions to split citation signals.&lt;/li&gt;
&lt;li&gt;Marketing hero pages with vague taglines and no concrete factual headings.&lt;/li&gt;
&lt;/ul&gt;




&lt;h3&gt;
  
  
  Audit Any URL in 10 Seconds (Free &amp;amp; Open Source)
&lt;/h3&gt;

&lt;p&gt;Because I got tired of running &lt;code&gt;curl&lt;/code&gt;, checking seven &lt;code&gt;robots.txt&lt;/code&gt; user-agents, inspecting raw HTML word counts, and validating &lt;code&gt;JSON-LD&lt;/code&gt; by hand, I built &lt;strong&gt;GEO-Audit&lt;/strong&gt; — a zero-dependency Node.js CLI and web auditor.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Option A — Test in your browser (no signup):&lt;/strong&gt;&lt;br&gt;
Paste any URL at &lt;a href="https://geo-audit-c8g.pages.dev" rel="noopener noreferrer"&gt;https://geo-audit-c8g.pages.dev&lt;/a&gt; to get a 0–100 score across all 5 pillars plus an instant downloadable starter &lt;code&gt;llms.txt&lt;/code&gt; draft.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Option B — Run locally with the MIT CLI (Node 20+, zero &lt;code&gt;npm install&lt;/code&gt;):&lt;/strong&gt;&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;git clone https://github.com/guyamos99/geo-audit
cd geo-audit
node cli/geo-audit.mjs https://yoursite.com
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;It checks:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;All 7 major AI crawler rules in &lt;code&gt;robots.txt&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/llms.txt&lt;/code&gt; &amp;amp; &lt;code&gt;/llms-full.txt&lt;/code&gt; presence + spec compliance (including SPA HTML 404 trap detection)&lt;/li&gt;
&lt;li&gt;SSR / raw HTML readability vs. JS-only shells&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;JSON-LD&lt;/code&gt; validity and entity coverage&lt;/li&gt;
&lt;li&gt;Citability &amp;amp; metadata signals&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If you maintain a large docs site or run client audits and want to automate this across up to 500 pages locally — with auto-generated &lt;code&gt;llms.txt&lt;/code&gt; + &lt;code&gt;llms-full.txt&lt;/code&gt;, white-label printable HTML reports, and a GitHub Action that opens PRs when your score or docs change — there's also an optional &lt;strong&gt;$29 one-time Pro toolkit&lt;/strong&gt; (&lt;a href="https://amosphere817.gumroad.com/l/geo-audit-pro" rel="noopener noreferrer"&gt;https://amosphere817.gumroad.com/l/geo-audit-pro&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;Otherwise, the free CLI and web tool cover single-page checks completely. Run your site through it and drop your score (or any weird edge cases you hit) in the comments!&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
      <category>opensource</category>
      <category>ai</category>
    </item>
  </channel>
</rss>
