There's an assumption sitting quietly inside most engineering teams: if Google can crawl and render our site, AI crawlers can too.
It's wrong. And...
For further actions, you may consider blocking this person and/or reporting abuse
Does serving prerendered HTML only to bot user agents still work here, or does that get treated like cloaking? For a new build SSR is the obvious call, but for a big existing SPA a prerender layer just for crawlers looks like the cheapest way out, and I'm not sure whether Bing or the AI crawlers punish it.
The key distinction is whether the content is the same. Cloaking is when the bot and the human see something semantically different. Serving a bot the prerendered version of the same content a human gets after hydration isn't cloaking — Google explicitly allows it, because the content matches.
Where it breaks: the prerender and the client drift apart over time. The human sees an updated price, the bot has the old one cached. Now the content is different — and that's the problem, not by intent but by neglect.
On punishment — Bing officially supports dynamic rendering. Nobody's made promises about the AI crawlers, because they simply don't execute JS: prerendering isn't a special case for them, it's the only way they see anything at all.
But if you're starting a new project — don't touch dynamic rendering at all. It's an extra layer to maintain and keep in sync. SSR or SSG solve the same thing without the split. A prerender layer only earns its place when rewriting a large SPA costs more than living with the fragility.
The
curl | greptest is necessary but it quietly assumes the crawler gets served the same bytes you do, and that's the failure mode I'd add. Plenty of "SSR" setups sit behind a CDN or edge middleware that branches on User-Agent, Accept headers, or cookies — a Vercel edge function that redirects based on geo, a bot-detection layer that serves an interstitial, a cache that only warms the SSR variant for logged-in traffic. Your curl comes from a residential IP with a normal UA and passes. GPTBot arrives from a datacenter range withUser-Agent: GPTBotand gets a challenge page, a 403, or a stale cache entry that predates your last deploy.So the honest version of the 30-second test is to spoof the crawler:
curl -A "GPTBot" -s https://you.com/page | grep -c "ld+json", and ideally from something that isn't your office IP. The gap between "my content is in the HTML" and "the content this specific bot receives is in the HTML" is exactly where the quiet failures you're describing love to hide — same category of bug, one layer further out.You're right, and it's better than my test. curl with my UA and home IP checks that the content is in the HTML — but not that it'll be served to GPTBot from a datacenter. Those are two different questions and I collapsed them into one.
-A "GPTBot" is mandatory, I was wrong to leave it out. IP is harder from the command line, but anyone on Cloudflare has a shortcut: server logs. Filter by the AI bots' UA and look at the response codes. 200 — they see you. 403, a challenge, or a redirect to login — they don't, and no view-source will show it, because you're looking from a browser with a human fingerprint.
This is exactly the class of failure the article is about, one layer deeper: the HTML can be perfect and the bot still gets a challenge page from the CDN before it ever reaches the content. Allowing it in robots.txt doesn't save you here — the WAF fires before robots.txt is even consulted.
Thanks — this is worth pulling into the article itself.
Done — added a "One layer deeper: the CDN" section above, credited to you.
Update: this turned into its own post rather than a paragraph — the WAF layer had more in it than a reply could hold. Credited you at the top, since it started here. Thanks for the nudge.
This explains why some pages rank well but never show up in AI answers. view-source is still one of the most useful debugging tools.
Exactly. And the sneaky part is that view-source tells the truth while DevTools lies: the Elements panel has already rendered the DOM, so the markup shows up there even when it isn't in the source. People check with the wrong tool and walk away reassured.
the render gap is real but it's usually the second thing to check. a lot of sites blocked GPTBot back in 2023 during the training scrape panic, then never revisited, and OAI-SearchBot and ChatGPT-User inherit the same robots.txt vibes plus whatever your WAF decides. an empty shell and a 403 look identical in a chat answer, so people go rebuild rendering when the fetch never landed.
i'd grep the edge logs for those three user agents and see what status they actually got before touching the app.
for the no-rewrite path, edge middleware that serves a prerendered snapshot on bot UA has generally held up better than trying to bolt SSR onto an existing SPA router. cheap to revert, too.
That's why I had to make aihu to serve dual purpose audiences. Check out the concept.
Interesting angle - you're solving it at the framework layer instead of patching it after the fact. The @agent block emitting an MCP tool schema alongside the Web Component is the part I hadn't seen done: most of what I write about is retrofitting discoverability onto sites that were never built for it. Building the agent surface into the SFC means it can't drift out of sync with the component, which is exactly the failure mode I keep finding in the wild (schema that describes a page version from two deploys ago).
The agent-readiness plugin emitting llms.txt and the MCP Server Card automatically is the right instinct too - the manual version of that is where teams give up.
One thing I'd be curious about: since the output is vanilla custom elements, does the SSR path in @aihu/server put the rendered content in the initial HTML response, or does hydration still own it? That's the line where AI crawlers either see the page or don't - they don't execute JS, so anything the browser assembles post-load is invisible to them regardless of how good the agent manifest is.
didn't even think about the difference between googlebot and ai crawlers, ngl. wonder if updating the robots.txt for specific bots is enough to fix this...
Different layer, unfortunately.
robots.txt controls whether a bot is allowed to fetch your page. Rendering controls whether there's anything in the page once it does.
Allow GPTBot all you want - it'll happily fetch your empty
<div id="root">, just faster.Both are worth checking, and they fail independently. I've seen sites with a spotless robots.txt that are invisible because everything is client-rendered, and properly server-rendered sites that are invisible because Cloudflare quietly blocks AI crawlers at the CDN layer before robots.txt even enters the picture.
Permission first, then payload. Neither substitutes for the other.
(The robots.txt side has its own trap, by the way: every provider runs two bots. GPTBot trains the model, OAI-SearchBot fetches pages live for answers. Blocking one doesn't block the other, and per OpenAI's docs, opting out of OAI-SearchBot means you don't appear in ChatGPT search at all. Worth a separate look.)
Didn't know AI bots can't execute JavaScript files except Gemini.
Yeah, that one surprised me too when I first saw the Vercel/MERJ logs. The counterintuitive part is that they DO download the JS files — GPTBot fetches them in ~11% of requests — they just never run them. So it's not that they can't reach your scripts. They reach them, look at the raw text, and move on.
Gemini being the exception makes sense once you see why: it rides on Googlebot's rendering infrastructure, so it inherits the ability to execute JS. Everyone else built their crawler fresh and skipped the rendering step — it's expensive at scale.
Easy way to see it yourself: curl any client-rendered page and look at what actually comes back. That's roughly what most of them see.