Quick test before you read on. Run this against one of your own content pages:
curl -s https://yoursite.com/some-page | grep -i "a sentence from your main content"
If that sentence doesn't come back, then as far as most AI crawlers are concerned, it isn't on your page. And that's a bigger deal than it used to be.
Why this happens
Google spent a decade building a rendering pipeline: Googlebot fetches your HTML, queues the page, runs a headless Chrome render, executes your JavaScript, then indexes the result. It's expensive and slow, and Google eats that cost because search is its business.
The AI crawlers — GPTBot, ClaudeBot, PerplexityBot, and the rest — mostly don't do this. They fetch your raw HTML response and move on. No render queue, no JS execution. Rendering the whole web at scale isn't worth it to them when their goal is harvesting text.
The practical consequence is blunt: the AI web is the no-JavaScript web. A page that looks perfect in a browser can be an empty <div id="root"></div> to the crawler deciding whether you're worth citing.
Who this hits
- Client-side-rendered SPAs (plain CRA-style React, Vue, Angular without SSR): the worst case — crawlers see a shell.
- Content behind "load more" / infinite scroll: only the initial HTML is visible.
- JS-injected sections on otherwise-fine pages: the sneaky one. Your article renders server-side, but the pricing table you hydrate from an API — the part that answers the question — is invisible.
Frameworks with SSR/SSG (Next.js, Nuxt, SvelteKit) and classic server-rendered stacks (WordPress, Rails, etc.) are generally fine. WordPress, for all the jokes, is more AI-legible than a slick client-rendered SPA.
The three ways to check
- curl + grep (above) — is your text in the raw HTML?
- DevTools with JS disabled — reload and see what survives.
- Fetch it like a crawler does — pull the server HTML the way GPTBot would and read what a model can actually see. If there's too little readable text, you've found the problem: that's the crawler's experience of your page.
The fix, cheapest first
- Already server-rendered? Nothing to do — just verify the JS-injected widgets.
- On a framework that supports SSR/SSG? Turn it on. SSG for content pages is ideal: complete HTML at build time, fast for humans, fully legible to every crawler.
- Can't adopt SSR? Server-render at least the critical content — the H1, the opening answer, key facts — and let JS enhance from there.
-
Don't forget metadata: JS-injected
<title>, meta descriptions, and JSON-LD have the same problem. If a tag manager adds your structured data client-side, crawlers never see it. It belongs in the server-rendered<head>.
The one exception
Google AI Overviews inherit Googlebot's rendering, so JS content Google indexes can surface there. That's why some SPAs appear in AI Overviews while being totally absent from ChatGPT and Perplexity — different pipelines, different capabilities. Treat Google as the exception and raw HTML as the baseline.
Curious how many people here have actually checked — if you curl your own site, is your main content in the response? Genuinely interested in how common the SPA blind spot is in practice.
If you'd rather not check this by hand, I maintain a free AI-readiness audit that fetches your page's server-rendered HTML the way a crawler would and scores what's actually readable — no signup.
Top comments (12)
This is becoming a real web quality check. If the important content only exists after client-side work, humans may still see the page while crawlers, agents, and answer engines see a thin shell. The 30-second check is useful because it makes that failure visible fast.
Exactly — that's the gap I kept running into. The page looks complete in a browser because the JS runs, so it's easy to assume everything's fine. But GPTBot, ClaudeBot, and most answer-engine fetchers don't execute JavaScript the way a full browser does (or they do it inconsistently and on their own budget), so they often just get the initial HTML shell.
The tricky part is it's invisible until you look for it — your analytics, your Lighthouse score, your own eyes all say the page is fine. That's why I leaned on the "fetch the raw HTML and show what's actually there" approach: it makes the shell-vs-rendered gap obvious in one glance instead of you having to reason about hydration.
The fix usually isn't "abandon React" either — SSR/SSG or prerendering the critical content gets you most of the way there. The check is really just about knowing which bucket you're in.
That raw-HTML check is the right instinct because it removes the comforting browser illusion. If the first response is mostly a shell, then the page is asking crawlers and answer engines to do extra work before they can even understand the business.
The practical test I like is simple: if the title, offer, location, product facts, and primary content are not visible in the initial HTML, assume some important agents will miss or flatten them.
Well put, Alex — that checklist (title, offer, location, product facts, primary content) is basically a "does the raw HTML stand on its own?" test, and I like that it's concrete enough to actually run.
The one thing I'd add: it's not always binary. Even when content is in the initial HTML, agents can still flatten it if the structure is ambiguous — they get the words but lose the relationships (which number is the price, which line is the location, what's the product vs. a related item). That's where JSON-LD earns its keep. Putting your key facts — Product, LocalBusiness, Offer, price, address — into structured data in the raw HTML gives agents an unambiguous, machine-readable copy that survives even if they only skim the rendered text. So my rule of thumb ends up being two layers: get the critical content into the initial HTML and mirror the hard facts in schema, so there's no way for an agent to guess wrong.
Curious whether you lean on structured data for this, or prefer to keep the HTML itself clean enough that it isn't needed?
That is the practical line for me too. The rendered page can look perfect while the raw document is almost empty, and that creates a gap between what humans see and what crawlers or answer engines can reliably extract.
I would treat the raw HTML check like a smoke test. It does not prove the site is great, but it quickly reveals when the most important business facts are hiding behind client-side timing.
"Smoke test" is the perfect way to frame it, Alex — cheap to run, doesn't prove excellence, but instantly exposes the failure mode that matters. And you nailed the core problem in one line: the gap between what humans see and what machines can reliably extract. That gap is invisible precisely because everything looks fine.
Good conversation - this is the kind of thing more teams should be checking before they worry about anything fancier.
Yes, and the cheapness is the point. A smoke test does not need to prove the whole site is perfect for AI readers. It just needs to catch the embarrassing failure modes early: empty HTML, missing business facts, invisible offers, and client-only content.
That list is a good one to keep handy — empty HTML, missing business facts, invisible offers, client-only content. Four things, all cheap to check, and catching any one of them early saves a lot of confused "why aren't we showing up" analysis later.
Thanks for the back and forth on this, Alex.
@alexshev
Wrote this up properly - credited you, thanks for the framing.
dev.to/osama1malik/the-30-second-r...
Nice, glad it helped. The 30-second version is useful because it gives people a quick failure check before they over-engineer the solution. Empty HTML and missing business facts are boring problems, but they are exactly the kind that quietly kills AI visibility.
Appreciate the credit. I think that framing is worth spreading because it keeps the check practical: first prove the crawler or AI reader can see the page, then worry about deeper optimization.
That's a good way to put it - prove it can be seen, then optimize. Step one is
easy to skip because it feels too basic, and that's exactly the trap: you can
burn weeks on schema and content structure for a page nobody could read in the
first place.
Thanks again for the back and forth on this, Alex - the post ended up a lot
better than my original draft because of it.