There's an assumption sitting quietly inside most engineering teams: if Google can crawl and render our site, AI crawlers can too.
It's wrong. And it's the single most expensive wrong assumption in technical SEO right now.
The gap
Googlebot renders JavaScript. It has done for years — the page goes into a render queue, headless Chromium executes the scripts, the result gets indexed. Deferred, but it works.
AI crawlers don't render anything.
A server-log study by Vercel and MERJ, published 17 December 2024, found that none of the major AI crawlers execute JavaScript. That includes OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, and Anthropic's ClaudeBot. A separate analysis of over 500 million GPTBot fetches found zero evidence of JavaScript execution.
The detail I find most telling: these crawlers do download JavaScript files. GPTBot in roughly 11.5% of requests, ClaudeBot in roughly 23.84%. They fetch the scripts and never run them.
There's no partial credit here. Either your content is in the initial HTML response, or it isn't.
The one exception is Google Gemini, which rides on Googlebot's Web Rendering Service and inherits its ability to execute JS — along with all its queue delays.
So both things are true at once
Your React site ranks on page one. Google rendered it, indexed it, ranks it.
Then a customer asks ChatGPT the exact question your page answers, and you're not in the response. Nothing is wrong with your rankings. The crawler received an empty shell:
<body>
<div id="root"></div>
<script type="module" src="/assets/index-a1b2c3.js"></script>
</body>
You are simultaneously first and invisible.
The trap inside the trap: structured data
This is where it gets specific.
You build a React site. You add react-helmet. You inject JSON-LD through it — Organization, LocalBusiness, FAQPage, the full graph:
<Helmet>
<script type="application/ld+json">
{JSON.stringify(schema)}
</script>
</Helmet>
You open DevTools, expand <head>, and there it is. Beautiful markup. Job done.
It isn't done. DevTools shows you the rendered DOM — the page after JavaScript has run. That is not what a crawler receives.
Open view-source: instead. Not the Elements panel. If your JSON-LD only exists in DevTools and not in view-source, then GPTBot, ClaudeBot and PerplexityBot never see it.
Google will still find it, because Google renders. That's why this failure is so quiet — every tool you'd normally check with says you're fine.
One nuance worth knowing: content doesn't have to be visible prose to count. Data embedded in the initial HTML as inline JSON or server-rendered payload is in the raw response and readable. What's missed is specifically what the browser builds after load.
Why this compounds
Around 92% of ChatGPT agent queries rely on Bing's index — and Bingbot's JavaScript rendering is limited. So a client-side rendered SPA risks being absent from Bing and from every AI crawler that queries it. Two failures, one root cause.
The test takes thirty seconds
curl -s https://you.com/page | grep -c "ld+json"
curl -s https://you.com/page | grep "your headline text"
If grep returns nothing, neither does the crawler.
Do it on your pricing page, your service pages, your best article. Those are the pages with the most to lose.
One layer deeper: the CDN
Everything above assumes the crawler receives the same response you do. It might not.
The curl test above uses your User-Agent and your IP. A CDN or edge middleware can serve differently based on User-Agent, headers, or geography — and AI bots come from datacenter IPs with declared bot UAs, which is exactly the traffic a WAF is tuned to challenge. Your HTML can be perfect and GPTBot still gets a 403, a challenge page, or a stale cache entry before it ever reaches the content.
So test as the bot, not as yourself:
curl -A "GPTBot" -s https://you.com/page | grep -c "ld+json"
And if you're on Cloudflare, the honest check is your server logs: filter by the AI crawlers' User-Agents and read the response codes. 200 means they see you. 403, a redirect, or a challenge means they don't — and no view-source will reveal it, because you're looking from a browser with a human fingerprint.
robots.txt doesn't save you here either: the WAF fires before robots.txt is consulted. Permission and access are different layers, and both fail silently.
(This section came out of a sharp comment from Wren Calloway below — worth reading the thread.)
The fix is not new
Server-side rendering, static generation, or prerendering. Next.js, Nuxt, Angular Universal, Django SSR — the framework matters less than the principle: put the real content in the HTML the server sends.
On our own platform the JSON-LD is rendered directly into the DOM rather than injected through Helmet, and key pages are prerendered. Not because it's elegant — because it's the difference between having schema and having schema anyone can read.
This is the same advice that's been correct for ten years. What changed is the cost of ignoring it. It used to mean waiting on Google's render queue. Now it means being absent from answers entirely, in the fastest-growing discovery channel there is.
Related reading:
Top comments (14)
Does serving prerendered HTML only to bot user agents still work here, or does that get treated like cloaking? For a new build SSR is the obvious call, but for a big existing SPA a prerender layer just for crawlers looks like the cheapest way out, and I'm not sure whether Bing or the AI crawlers punish it.
The key distinction is whether the content is the same. Cloaking is when the bot and the human see something semantically different. Serving a bot the prerendered version of the same content a human gets after hydration isn't cloaking — Google explicitly allows it, because the content matches.
Where it breaks: the prerender and the client drift apart over time. The human sees an updated price, the bot has the old one cached. Now the content is different — and that's the problem, not by intent but by neglect.
On punishment — Bing officially supports dynamic rendering. Nobody's made promises about the AI crawlers, because they simply don't execute JS: prerendering isn't a special case for them, it's the only way they see anything at all.
But if you're starting a new project — don't touch dynamic rendering at all. It's an extra layer to maintain and keep in sync. SSR or SSG solve the same thing without the split. A prerender layer only earns its place when rewriting a large SPA costs more than living with the fragility.
The
curl | greptest is necessary but it quietly assumes the crawler gets served the same bytes you do, and that's the failure mode I'd add. Plenty of "SSR" setups sit behind a CDN or edge middleware that branches on User-Agent, Accept headers, or cookies — a Vercel edge function that redirects based on geo, a bot-detection layer that serves an interstitial, a cache that only warms the SSR variant for logged-in traffic. Your curl comes from a residential IP with a normal UA and passes. GPTBot arrives from a datacenter range withUser-Agent: GPTBotand gets a challenge page, a 403, or a stale cache entry that predates your last deploy.So the honest version of the 30-second test is to spoof the crawler:
curl -A "GPTBot" -s https://you.com/page | grep -c "ld+json", and ideally from something that isn't your office IP. The gap between "my content is in the HTML" and "the content this specific bot receives is in the HTML" is exactly where the quiet failures you're describing love to hide — same category of bug, one layer further out.You're right, and it's better than my test. curl with my UA and home IP checks that the content is in the HTML — but not that it'll be served to GPTBot from a datacenter. Those are two different questions and I collapsed them into one.
-A "GPTBot" is mandatory, I was wrong to leave it out. IP is harder from the command line, but anyone on Cloudflare has a shortcut: server logs. Filter by the AI bots' UA and look at the response codes. 200 — they see you. 403, a challenge, or a redirect to login — they don't, and no view-source will show it, because you're looking from a browser with a human fingerprint.
This is exactly the class of failure the article is about, one layer deeper: the HTML can be perfect and the bot still gets a challenge page from the CDN before it ever reaches the content. Allowing it in robots.txt doesn't save you here — the WAF fires before robots.txt is even consulted.
Thanks — this is worth pulling into the article itself.
Done — added a "One layer deeper: the CDN" section above, credited to you.
Update: this turned into its own post rather than a paragraph — the WAF layer had more in it than a reply could hold. Credited you at the top, since it started here. Thanks for the nudge.
This explains why some pages rank well but never show up in AI answers. view-source is still one of the most useful debugging tools.
Exactly. And the sneaky part is that view-source tells the truth while DevTools lies: the Elements panel has already rendered the DOM, so the markup shows up there even when it isn't in the source. People check with the wrong tool and walk away reassured.
the render gap is real but it's usually the second thing to check. a lot of sites blocked GPTBot back in 2023 during the training scrape panic, then never revisited, and OAI-SearchBot and ChatGPT-User inherit the same robots.txt vibes plus whatever your WAF decides. an empty shell and a 403 look identical in a chat answer, so people go rebuild rendering when the fetch never landed.
i'd grep the edge logs for those three user agents and see what status they actually got before touching the app.
for the no-rewrite path, edge middleware that serves a prerendered snapshot on bot UA has generally held up better than trying to bolt SSR onto an existing SPA router. cheap to revert, too.
That's why I had to make aihu to serve dual purpose audiences. Check out the concept.
Interesting angle - you're solving it at the framework layer instead of patching it after the fact. The @agent block emitting an MCP tool schema alongside the Web Component is the part I hadn't seen done: most of what I write about is retrofitting discoverability onto sites that were never built for it. Building the agent surface into the SFC means it can't drift out of sync with the component, which is exactly the failure mode I keep finding in the wild (schema that describes a page version from two deploys ago).
The agent-readiness plugin emitting llms.txt and the MCP Server Card automatically is the right instinct too - the manual version of that is where teams give up.
One thing I'd be curious about: since the output is vanilla custom elements, does the SSR path in @aihu/server put the rendered content in the initial HTML response, or does hydration still own it? That's the line where AI crawlers either see the page or don't - they don't execute JS, so anything the browser assembles post-load is invisible to them regardless of how good the agent manifest is.
didn't even think about the difference between googlebot and ai crawlers, ngl. wonder if updating the robots.txt for specific bots is enough to fix this...
Different layer, unfortunately.
robots.txt controls whether a bot is allowed to fetch your page. Rendering controls whether there's anything in the page once it does.
Allow GPTBot all you want - it'll happily fetch your empty
<div id="root">, just faster.Both are worth checking, and they fail independently. I've seen sites with a spotless robots.txt that are invisible because everything is client-rendered, and properly server-rendered sites that are invisible because Cloudflare quietly blocks AI crawlers at the CDN layer before robots.txt even enters the picture.
Permission first, then payload. Neither substitutes for the other.
(The robots.txt side has its own trap, by the way: every provider runs two bots. GPTBot trains the model, OAI-SearchBot fetches pages live for answers. Blocking one doesn't block the other, and per OpenAI's docs, opting out of OAI-SearchBot means you don't appear in ChatGPT search at all. Worth a separate look.)
Didn't know AI bots can't execute JavaScript files except Gemini.
Yeah, that one surprised me too when I first saw the Vercel/MERJ logs. The counterintuitive part is that they DO download the JS files — GPTBot fetches them in ~11% of requests — they just never run them. So it's not that they can't reach your scripts. They reach them, look at the raw text, and move on.
Gemini being the exception makes sense once you see why: it rides on Googlebot's rendering infrastructure, so it inherits the ability to execute JS. Everyone else built their crawler fresh and skipped the rendering step — it's expensive at scale.
Easy way to see it yourself: curl any client-rendered page and look at what actually comes back. That's roughly what most of them see.