Quick test before you read on. Run this against one of your own content pages:
bash
curl -s https://yoursite.com/some-page | grep -i "a sentence from your main content"
If that sentence doesn't come back, then as far as most AI crawlers are concerned, it isn't on your page. And that's a bigger deal than it used to be.
Why this happens
Google spent a decade building a rendering pipeline: Googlebot fetches your HTML, queues the page, runs a headless Chrome render, executes your JavaScript, then indexes the result. It's expensive and slow, and Google eats that cost because search is its business.
The AI crawlers — GPTBot, ClaudeBot, PerplexityBot, and the rest — mostly don't do this. They fetch your raw HTML response and move on. No render queue, no JS execution. Rendering the whole web at scale isn't worth it to them when their goal is harvesting text.
The practical consequence is blunt: the AI web is the no-JavaScript web. A page that looks perfect in a browser can be an empty <div id="root"></div> to the crawler deciding whether you're worth citing.
Who this hits
Client-side-rendered SPAs (plain CRA-style React, Vue, Angular without SSR): the worst case — crawlers see a shell.
Content behind "load more" / infinite scroll: only the initial HTML is visible.
JS-injected sections on otherwise-fine pages: the sneaky one. Your article renders server-side, but the pricing table you hydrate from an API — the part that answers the question — is invisible.
Frameworks with SSR/SSG (Next.js, Nuxt, SvelteKit) and classic server-rendered stacks (WordPress, Rails, etc.) are generally fine. WordPress, for all the jokes, is more AI-legible than a slick client-rendered SPA.
The three ways to check
curl + grep (above) — is your text in the raw HTML?
DevTools with JS disabled — reload and see what survives.
Fetch it like a crawler does — I built a free AI Snippet Simulator (no signup) that pulls your server HTML the way GPTBot would and reports what a model can actually read and cite. If it says "too little readable text," you've found the problem — that verdict is the crawler's experience of your page.
The fix, cheapest first
Already server-rendered? Nothing to do — just verify the JS-injected widgets.
On a framework that supports SSR/SSG? Turn it on. SSG for content pages is ideal: complete HTML at build time, fast for humans, fully legible to every crawler.
Can't adopt SSR? Server-render at least the critical content — the H1, the opening answer, key facts — and let JS enhance from there.
Don't forget metadata: JS-injected <title>, meta descriptions, and JSON-LD have the same problem. If a tag manager adds your structured data client-side, crawlers never see it. It belongs in the server-rendered <head>.
The one exception
Google AI Overviews inherit Googlebot's rendering, so JS content Google indexes can surface there. That's why some SPAs appear in AI Overviews while being totally absent from ChatGPT and Perplexity — different pipelines, different capabilities. Treat Google as the exception and raw HTML as the baseline.
Curious how many people here have actually checked — if you curl your own site, is your main content in the response? Genuinely interested in how common the SPA blind spot is in practice.
*(Leave `canonical_url` blank so dev.to is canonical; you still get the in-body link + referral + dev.to's own ranking. Tags: `webdev, ai, javascript, seo` — the `javascript` tag reaches a big, different audience than your first post.)*
---
## Reddit post (fresh angle: training vs search — for r/Agentic_SEO)
Post this in **r/Agentic_SEO** (the sub where you found that build post — it's smaller, on-topic, and friendlier to this than r/TechSEO's shill filter). Keep it link-free; share the tool only if asked.
**Title:** PSA: blocking GPTBot doesn't remove you from ChatGPT — they're completely different crawlers
**Body:**
keep seeing people add `User-agent: GPTBot / Disallow: /` and think they've "blocked ChatGPT." they haven't, and it's worth understanding why.
OpenAI runs (at least) three separate crawlers:
- **GPTBot** — training. this is the one everyone blocks.
- **OAI-SearchBot** — builds the index behind ChatGPT's *search* answers.
- **ChatGPT-User** — fetches a page live when a user asks ChatGPT about it.
so blocking GPTBot opts you out of model *training*, but you're still fully visible in ChatGPT's search answers and live browsing. two different pipelines, two different user agents. i keep finding sites that think they've disappeared from ChatGPT when they're still getting cited in it daily.
the flip side is the more common mistake imo: people do a blanket "block all AI" and accidentally kill OAI-SearchBot + PerplexityBot — which are the ones that actually *send traffic*. so they've de-indexed themselves from the AI answers while keeping the training crawlers that give nothing back.
and it's not just OpenAI — Anthropic splits ClaudeBot (training) from Claude-SearchBot (search), Google splits Google-Extended (Gemini training, a robots token not even a crawler) from Googlebot, etc. almost every company separates "train on me" from "cite me," which means you can allow one and block the other on purpose.
how's everyone here thinking about this split? blocking training but keeping search open? all-or-nothing? i keep going back and forth for content-heavy sites and would love to hear how others land on it.
*(If someone asks "is there a list of all these?" → then reply with the directory link + "full disclosure it's mine.")*
---
**One process note:** the training-vs-search insight now appears in your dev.to piece, your Reddit post, *and* your site's guides — that's fine and deliberate (it's the single most useful counterintuitive fact in this niche, and repetition across channels is how a point sticks), but don't reuse the *exact same wording* on-site vs off-site, which these don't. Go reply to those PH comments first — that's the time-sensitive one.
Top comments (1)
This is becoming a real web quality check. If the important content only exists after client-side work, humans may still see the page while crawlers, agents, and answer engines see a thin shell. The 30-second check is useful because it makes that failure visible fast.