DEV Community

Jangwook Kim
Jangwook Kim

Posted on • Originally published at jangwook.net

AI Crawlers Don't Run Your JavaScript

#geo #seo #ssr #aicrawler

Here's the short version: if your site renders its content in the browser with JavaScript, most AI crawlers can't see it. Not "see it late" or "see it partially." They fetch your raw HTML, pull the text, and leave. GPTBot, ClaudeBot, and PerplexityBot don't render your page, and they don't wait around for a second pass.

That matters more every month. People ask ChatGPT, search on Perplexity, and read Google's AI Overview before they ever hit a blue link. The crawlers that build those answers behave nothing like Googlebot. So the old comfort ("Googlebot handles JS fine, we're covered") quietly stops being true. I didn't want to just assert this, so I reproduced it in a sandbox with curl.

What AI crawlers receive as raw HTML — CSR is an empty shell, SSR carries the content

Two meanings of "rendering," sorted out first

To read the rest of this cleanly, pin down where your content actually becomes HTML.

With server-side rendering (SSR) and static generation (SSG), the server sends finished HTML. Whether a browser or a crawler receives it, the response already contains <h1>Business Name</h1>, the address, and the body copy. With client-side rendering (CSR), the server ships a near-empty shell (<div id="app"></div>) plus a JavaScript bundle. The real content gets filled in when the browser runs that JS. A typical React or Vue SPA works this way.

A human in a browser never notices the difference, because browsers execute JavaScript. The gap only shows up for a visitor that doesn't run JS. And the web now has a fast-growing population of exactly those visitors: AI crawlers.

Don't put Googlebot and AI crawlers in the same box

This is where most people slip. Google's own docs (Understand JavaScript SEO Basics) explain that Googlebot renders pages with a headless Chromium, executes the JavaScript, and indexes the rendered HTML. Google has long called dynamic rendering "a workaround and not a recommended solution," steering you toward SSR, SSG, or hydration instead (Dynamic Rendering as a workaround). In March 2026 they even removed the old warning about keeping pages functional without JavaScript. Google trusts its own renderer that much.

Reading that as "CSR is safe everywhere now" will burn you. Those docs are about Googlebot, not about AI crawlers. From what I've verified, and consistent with industry crawl-data analysis (Vercel, "The rise of the AI crawler"; reference, not official), GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Bytespider don't render JS at all. One analysis of over 500 million GPTBot fetches reportedly found zero evidence of JavaScript execution (reference, not official). GPTBot may download JS files, but it doesn't run them.

There's one exception worth naming. Google Gemini leans on Googlebot's Web Rendering Service, so it can execute JavaScript. That means Google's AI Overviews might see a CSR page. ChatGPT, Claude, and Perplexity won't. So don't generalize "an AI read my SPA" from a single Google data point.

One line of curl reproduces exactly what a crawler sees

Talk is cheap, so I measured it. The trick is simple: curl doesn't execute JavaScript. That makes it a faithful stand-in for a non-rendering AI crawler fetching your raw HTML off the server.

I built two versions of a fictional cafe site in a sandbox. Identical content. One CSR, one SSR.

<!-- csr.html — content injected only on the client -->
<div id="app"><p>Loading…</p></div>
<script>
  fetch('/data.json').then(r => r.json()).then(d => {
    document.getElementById('app').innerHTML =
      '<h1>' + d.name + '</h1><p>' + d.tagline + '</p>' +
      '<address>' + d.address + '</address>' +
      '<p>Signature: ' + d.signature + '</p>';
  });
</script>
Enter fullscreen mode Exit fullscreen mode 
<!-- ssr.html — the server response already carries the content -->
<main id="app">
  <h1>Aria Coffee Roasters</h1>
  <p>Single-origin specialty coffee, roasted in-house</p>
  <address>12 Somewhere-ro, Mapo-gu, Seoul</address>
  <p>Signature: Geisha hand-drip</p>
</main>
Enter fullscreen mode Exit fullscreen mode

Then I fetched both while spoofing GPTBot. Changing the User-Agent is cosmetic here; curl wasn't going to run the JS anyway.

curl -A "GPTBot/1.2" http://127.0.0.1:8971/csr.html | grep -c "Geisha"
# → 0

curl -A "GPTBot/1.2" http://127.0.0.1:8971/ssr.html | grep -c "Geisha"
# → 1
Enter fullscreen mode Exit fullscreen mode

The signature drink is exactly the detail that would get this cafe cited in an answer. It shows up zero times in the CSR response. Only the business name in the <title> survives; the body, address, and menu all vanish. Extract the readable text a crawler would keep, and the gap gets starker.

Crawler-extracted text via curl — CSR yields 29 chars, SSR yields 107

From the CSR page, the crawler salvaged "Aria Coffee Roasters Loading…". It kept 29 characters, loading spinner and all. The SSR page gave up 107 characters: name, description, address, signature menu, intact. Same content, same design, indistinguishable to a human. What the crawler reads is night and day.

So which one is your site? A 30-second check

It sounds like someone else's problem until you check, and checking takes half a minute. Two ways.

First, look at the crawler's view straight from your terminal. Drop in your URL and a phrase that must be on the page.

curl -A "GPTBot" https://example.com/my-page | grep "your key phrase"
Enter fullscreen mode Exit fullscreen mode

If that phrase isn't in the output, AI crawlers can't see it either. If only your title tag comes back and the body doesn't, you're leaning hard on CSR.

Second, disable JavaScript in DevTools and reload. In Chrome, open the command palette (Cmd+Shift+P) and run "Disable JavaScript." If the page goes blank or sticks on "Loading…," that's the screen GPTBot is looking at. This is the first thing I reach for when auditing a client site. No report needed; you can call it by eye.

How to fix it, by framework

The fix isn't novel: put your core content in the server's HTML response. Where you touch it depends on your stack.

  • Next.js: move data fetching to the server via App Router server components (RSC), getServerSideProps, or static generation. Don't fetch content only inside a useEffect.
  • Nuxt: universal mode is the default. Confirm ssr: true is still on and that the component in question isn't wrapped in <ClientOnly>.
  • Astro: static generation by default, so usually safe. Watch out for text that lives only inside a client:only island; it won't be in the initial HTML.
  • SvelteKit / Angular: turn on server execution in SvelteKit's load functions, or Angular Universal (SSR) for Angular.

One pattern deserves special caution: injecting structured data (JSON-LD) or meta tags through a client script like Google Tag Manager. It looks fine to a human, but AI crawlers never run that script, so the JSON-LD disappears with it. I measured this trap in why server-side beats JS for LocalBusiness structured data, and in the AI-crawler era that "server-side is more reliable" principle carries a lot more weight. Even a well-formed JSON-LD @graph entity model only pays off if it's in the server response.

If a full SSR migration feels heavy, hybrid is fine. Render the shell and the core text on the server, then hydrate only the interactive widgets on the client. There's a single test that matters: is the meaningful body text in the initial HTML?

Once you've fixed it, re-run the same curl command. Deploy pipelines skip prerender steps, CDNs cache a different response for bots, and specific routes quietly keep rendering client-side. I pick a few key landing pages and keep curl -A "GPTBot" ... | grep in the post-deploy checklist. One line catches the regression.

Worth adding: SSR helps Googlebot too. Googlebot does run JS, but it crawls and renders on separate queues. A CSR page goes through two stages (grab the HTML, then re-render later when rendering resources free up) that can delay when content lands in the index. Serve finished HTML and that render-queue wait disappears. AI crawler coverage is the goal; fresher indexing is the side benefit.

The llms.txt "fix" is oversold

Raise this topic and someone always says, "so just drop an llms.txt, right?" I think selling llms.txt as the cure for a CSR problem points in the wrong direction.

llms.txt is a community proposal to hand crawlers a Markdown summary of your site's content. The idea isn't bad. The reality is the issue. Google has flatly said it doesn't support llms.txt and has no plans to (Search Central Live, July 2025, Gary Illyes), and John Mueller likened it to the keywords meta tag that search engines have ignored for over a decade. It's a file where the site owner asserts what the site is about, which makes it easy to game. No major AI service has officially confirmed using it during inference. Adoption sat around 10% in a 300,000-domain study, and roughly 97% of valid llms.txt files reportedly received zero requests during May 2026 (reference, not official).

The point: the root reason an AI crawler can't read you isn't a missing summary file — it's body content hidden behind JavaScript. An llms.txt paves a detour around the cause instead of removing it. How to control AI-crawler access in the first place is its own topic, which I covered in controlling AI crawlers with robots.txt, so use that for allow/block policy. But step one of "get cited" is always server-side visibility.

Limits, stated honestly

Two things I want to be clear about.

First, this experiment reproduced a non-rendering fetch with curl; it didn't capture live GPTBot traffic. But the mechanism I reproduced, not executing JavaScript, is these crawlers' documented behavior, so the direction of the result holds.

Second, and more important: making your content visible with SSR does not guarantee a citation or a ranking. Visibility is necessary, not sufficient. Once a crawler can read you, content quality, trust, and structured data do the rest. It's the same reasoning behind Google's repeated line that structured data doesn't guarantee rankings. All this post promises is the first step: turning invisible into visible. What comes after is the content's job.

My takeaway is blunt. If you actually care about AI search, run curl -A "GPTBot" before you layer on any fancy GEO tactic. If your key phrase isn't in that output, every other optimization is being built on top of a blank page.

If you want structured data delivered reliably server-side, or a check on whether an existing SPA or headless setup is actually exposed to AI search and crawlers, I take on consulting and implementation work personally. The contact path on my profile is the easiest way to reach me.

Top comments (0)