There's an assumption sitting quietly inside most engineering teams: if Google can crawl and render our site, AI crawlers can too.
It's wrong. And it's the single most expensive wrong assumption in technical SEO right now.
The gap
Googlebot renders JavaScript. It has done for years — the page goes into a render queue, headless Chromium executes the scripts, the result gets indexed. Deferred, but it works.
AI crawlers don't render anything.
A server-log study by Vercel and MERJ, published 17 December 2024, found that none of the major AI crawlers execute JavaScript. That includes OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, and Anthropic's ClaudeBot. A separate analysis of over 500 million GPTBot fetches found zero evidence of JavaScript execution.
The detail I find most telling: these crawlers do download JavaScript files. GPTBot in roughly 11.5% of requests, ClaudeBot in roughly 23.84%. They fetch the scripts and never run them.
There's no partial credit here. Either your content is in the initial HTML response, or it isn't.
The one exception is Google Gemini, which rides on Googlebot's Web Rendering Service and inherits its ability to execute JS — along with all its queue delays.
So both things are true at once
Your React site ranks on page one. Google rendered it, indexed it, ranks it.
Then a customer asks ChatGPT the exact question your page answers, and you're not in the response. Nothing is wrong with your rankings. The crawler received an empty shell:
<body>
<div id="root"></div>
<script type="module" src="/assets/index-a1b2c3.js"></script>
</body>
You are simultaneously first and invisible.
The trap inside the trap: structured data
This is where it gets specific.
You build a React site. You add react-helmet. You inject JSON-LD through it — Organization, LocalBusiness, FAQPage, the full graph:
<Helmet>
<script type="application/ld+json">
{JSON.stringify(schema)}
</script>
</Helmet>
You open DevTools, expand <head>, and there it is. Beautiful markup. Job done.
It isn't done. DevTools shows you the rendered DOM — the page after JavaScript has run. That is not what a crawler receives.
Open view-source: instead. Not the Elements panel. If your JSON-LD only exists in DevTools and not in view-source, then GPTBot, ClaudeBot and PerplexityBot never see it.
Google will still find it, because Google renders. That's why this failure is so quiet — every tool you'd normally check with says you're fine.
One nuance worth knowing: content doesn't have to be visible prose to count. Data embedded in the initial HTML as inline JSON or server-rendered payload is in the raw response and readable. What's missed is specifically what the browser builds after load.
Why this compounds
Around 92% of ChatGPT agent queries rely on Bing's index — and Bingbot's JavaScript rendering is limited. So a client-side rendered SPA risks being absent from Bing and from every AI crawler that queries it. Two failures, one root cause.
The test takes thirty seconds
curl -s https://you.com/page | grep -c "ld+json"
curl -s https://you.com/page | grep "your headline text"
If grep returns nothing, neither does the crawler.
Do it on your pricing page, your service pages, your best article. Those are the pages with the most to lose.
The fix is not new
Server-side rendering, static generation, or prerendering. Next.js, Nuxt, Angular Universal, Django SSR — the framework matters less than the principle: put the real content in the HTML the server sends.
On our own platform the JSON-LD is rendered directly into the DOM rather than injected through Helmet, and key pages are prerendered. Not because it's elegant — because it's the difference between having schema and having schema anyone can read.
This is the same advice that's been correct for ten years. What changed is the cost of ignoring it. It used to mean waiting on Google's render queue. Now it means being absent from answers entirely, in the fastest-growing discovery channel there is.
Related reading:
Top comments (2)
didn't even think about the difference between googlebot and ai crawlers, ngl. wonder if updating the robots.txt for specific bots is enough to fix this...
Different layer, unfortunately.
robots.txt controls whether a bot is allowed to fetch your page. Rendering controls whether there's anything in the page once it does. Allow GPTBot all you want — it'll happily fetch your empty
, just faster.Both are worth checking, and they fail independently. I've seen sites with a spotless robots.txt that are invisible because everything is client-rendered, and properly server-rendered sites that are invisible because Cloudflare quietly blocks AI crawlers at the CDN layer before robots.txt even enters the picture.
Permission first, then payload. Neither substitutes for the other.
(The robots.txt side has its own trap, by the way: every provider runs two bots. GPTBot trains the model, OAI-SearchBot fetches pages live for answers. Blocking one doesn't block the other, and per OpenAI's docs, opting out of OAI-SearchBot means you don't appear in ChatGPT search at all. Worth a separate look.)