DEV Community

Cover image for Is Your React App Invisible to ChatGPT? Test What AI Crawlers Actually See
Furqan Khalid
Furqan Khalid

Posted on AI-assisted

Is Your React App Invisible to ChatGPT? Test What AI Crawlers Actually See

#react #javascript #nextjs #seo

Here's a bug report you'll never get: "ChatGPT says our product doesn't have a free tier."

Nobody files it, because nobody connects the dots. But if your pricing page is a client-rendered React app, there's a decent chance the AI system that answered that question received this:

<!doctype html>
<html>
  <head><title>Pricing | Acme</title></head>
  <body>
    <div id="root"></div>
    <script src="/assets/index-8f3a.js"></script>
  </body>
</html>
Enter fullscreen mode Exit fullscreen mode

Then it filled the gap from somewhere else: an old review, a competitor's comparison page, or a guess.

Why this is worse in 2026 than it was in 2020

Google spent years building a JavaScript rendering pipeline. Googlebot runs your JS (eventually), so client-side rendering became "mostly fine for SEO."

AI systems didn't inherit that pipeline. There are now three kinds of AI fetchers hitting your site:

Type Examples Renders JS?
Training crawlers GPTBot, ClaudeBot, CCBot Generally treated as no. Assume raw HTML.
AI search crawlers OAI-SearchBot, Claude-SearchBot, PerplexityBot Not reliably. Assume raw HTML.
Live user fetches ChatGPT-User, Claude-User, Perplexity-User Varies by product and changes over time. Don't count on it.

Vendors don't publish rendering guarantees, and behaviour changes. The only safe assumption is the one that has always been safe: if it's not in the HTML response, it might not exist.

Google's AI features (AI Overviews, AI Mode) use Google's index, which is rendered, so a CSR site can show up there and still be invisible to ChatGPT. That's exactly the kind of gap that's easy to miss.

Three kinds of AI fetcher compared with Googlebot and whether each renders JavaScript

Test it: raw vs rendered, side by side

This script fetches a URL twice, once as a plain HTTP request (what a non-rendering crawler gets) and once in headless Chromium (what a browser gets), then compares the text.

npm i playwright cheerio
npx playwright install chromium
Enter fullscreen mode Exit fullscreen mode 
// render-gap.mjs
import { chromium } from "playwright";
import * as cheerio from "cheerio";

const url = process.argv[2];
const mustContain = process.argv.slice(3); // phrases that MUST be visible to crawlers

function textOf(html) {
  const $ = cheerio.load(html);
  $("script, style, noscript, template, svg").remove();
  return $("body").text().replace(/\s+/g, " ").trim();
}

// 1. Raw HTML, as a non-rendering crawler would see it
const rawRes = await fetch(url, {
  headers: { "User-Agent": "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" },
});
const raw = await rawRes.text();

// 2. Rendered DOM after JavaScript
const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto(url, { waitUntil: "networkidle" });
const rendered = await page.content();
await browser.close();

const rawText = textOf(raw);
const renderedText = textOf(rendered);
const ratio = rawText.length / Math.max(renderedText.length, 1);

console.log(`Status (raw fetch):  ${rawRes.status}`);
console.log(`Raw text:            ${rawText.length.toLocaleString()} chars`);
console.log(`Rendered text:       ${renderedText.length.toLocaleString()} chars`);
console.log(`Raw / rendered:      ${(ratio * 100).toFixed(0)}%`);

const $raw = cheerio.load(raw);
console.log(`<title> in raw:      ${$raw("title").text() || "MISSING"}`);
console.log(`meta description:    ${$raw('meta[name="description"]').attr("content") ? "yes" : "MISSING"}`);
console.log(`<h1> in raw:         ${$raw("h1").first().text().trim() || "MISSING"}`);
console.log(`JSON-LD in raw:      ${$raw('script[type="application/ld+json"]').length} block(s)`);

for (const phrase of mustContain) {
  const inRaw = rawText.toLowerCase().includes(phrase.toLowerCase());
  console.log(`${inRaw ? "✅" : "❌"} "${phrase}" ${inRaw ? "is" : "is NOT"} in the raw HTML`);
}

if (ratio < 0.5) {
  console.log("\n⚠️  Less than half of the visible text exists without JavaScript.");
  process.exitCode = 1;
}
Enter fullscreen mode Exit fullscreen mode 
node render-gap.mjs https://your-site.com/pricing "Free plan" "per seat" "SOC 2"
Enter fullscreen mode Exit fullscreen mode 
Status (raw fetch):  200
Raw text:            142 chars
Rendered text:       6,418 chars
Raw / rendered:      2%
<title> in raw:      Pricing | Acme
meta description:    MISSING
<h1> in raw:         MISSING
JSON-LD in raw:      0 block(s)
❌ "Free plan" is NOT in the raw HTML
❌ "per seat" is NOT in the raw HTML
❌ "SOC 2" is NOT in the raw HTML

⚠️  Less than half of the visible text exists without JavaScript.
Enter fullscreen mode Exit fullscreen mode

The mustContain phrases are the important part. Pick the facts you'd be upset to see an AI get wrong (price, free tier, integrations, compliance) and make them a test.

Also check the status code. If the raw fetch with an AI user agent returns 403 or a challenge page while your browser gets 200, your problem is the CDN or WAF, not rendering. Look at your bot-management rules before changing any code.

The fixes, by framework

Next.js (App Router)

Server Components render on the server by default, so most App Router pages are already fine. The usual culprits:

// ❌ Whole page opted into client rendering, data fetched in useEffect
"use client";
export default function Pricing() {
  const [plans, setPlans] = useState([]);
  useEffect(() => { fetch("/api/plans").then(r => r.json()).then(setPlans); }, []);
  return <PlanTable plans={plans} />;
}
Enter fullscreen mode Exit fullscreen mode 
// ✅ Server Component fetches data; only the interactive toggle is a client component
import { BillingToggle } from "./billing-toggle"; // "use client" inside

export const revalidate = 3600; // static, refreshed hourly

export default async function Pricing() {
  const plans = await getPlans();
  return (
    <main>
      <h1>Pricing</h1>
      <BillingToggle />
      <PlanTable plans={plans} />
    </main>
  );
}
Enter fullscreen mode Exit fullscreen mode

Also watch for dynamic(() => import(...), { ssr: false }) around content (it's fine for widgets, not for text), and for content that only appears inside tabs or accordions that mount on click. Render it all and hide it with CSS instead.

Vite + React SPA

You have three options, from least to most work:

  1. Prerender the routes that matter at build time (marketing pages, docs, pricing), for example with vite-plugin-prerender or a small Playwright script that saves rendered HTML for a list of routes into dist/.
  2. Move the marketing site to a static-first framework (Astro, Next.js static export) and keep the SPA for the logged-in app at app.example.com.
  3. Adopt SSR (React Router framework mode, TanStack Start) if most of the product is public content.

For most teams, option 2 is the right answer: the app doesn't need to be crawlable, and the marketing site does.

Astro, Eleventy, Hugo

Static by default. Just be careful with client-only islands (client:only) that contain content rather than interactivity.

What about dynamic rendering?

Serving prerendered HTML only to bots (the old "dynamic rendering" pattern) still works technically, but Google describes it as a workaround, not a recommended solution. It also creates two versions of every page that can drift apart, and serving different content to bots is the definition of cloaking. Prefer SSR or static generation, which give everyone the same HTML.

Put the check in CI

Run the script against a preview deployment on every PR that touches marketing or docs routes:

# .github/workflows/render-gap.yml
name: Crawlability
on: pull_request
jobs:
  render-gap:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm ci && npx playwright install --with-deps chromium
      # PREVIEW_URL comes from your deploy provider's preview step
      - run: node scripts/render-gap.mjs "$PREVIEW_URL/pricing" "Free plan" "per seat"
      - run: node scripts/render-gap.mjs "$PREVIEW_URL/docs/quickstart" "npm install"
Enter fullscreen mode Exit fullscreen mode

That one check would have caught every "the AI said we don't have X" problem I've seen.

Beyond rendering

Server-side rendering gets your content into the HTML. Whether AI crawlers can reach it at all also depends on robots.txt, your CDN's bot rules, rate limits and status codes, and whether the page is indexed in Google and Bing. Vista AI's page audit checks those things together with the raw-vs-rendered gap for a URL. If you'd rather script it yourself, the render-gap.mjs above plus a robots.txt check covers the most common failures.

Run the script on your pricing page and share the raw/rendered percentage in the comments. Bonus points if it's under 10%.

Next in the series: How AI assistants decide which dev tool to recommend.

Top comments (0)