DEV Community

Cover image for AI crawlers don't run your JavaScript
richi.codes
richi.codes

Posted on Originally published at richi.codes

AI crawlers don't run your JavaScript

#ai #seo #rendering #javascript

This note has an interactive demo. Try it in the original

If your page builds its content with JavaScript in the browser, the crawlers behind ChatGPT, Claude and Perplexity see an empty shell. They download the HTML, they may even download your bundles, but they don't execute them. Googlebot does run JavaScript, which is why this goes unnoticed for so long: your page can rank fine in Google and still doesn't exist for a big part of AI search. The demo above puts both readers side by side, a real browser on the left and what a no-JS crawler gets on the right, with the same HTML going to both.

What the crawlers actually do

The clearest data I've found is Vercel's study of crawler traffic on its network, published in December 2024. Their conclusion is blunt: "none of the major AI crawlers currently render JavaScript". The list they checked includes OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic's ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. They also note that ChatGPT's and Claude's crawlers do fetch JavaScript files, they just don't run them, so client-side rendered content is invisible to them.

The exceptions are the crawlers that sit on top of a browser. According to the same study, Gemini uses Googlebot's infrastructure, which gives it full JavaScript rendering, and AppleBot renders through a browser-based crawler too.

And these aren't fringe bots. In the month Vercel measured, GPTBot made 569 million fetches and Claude's crawlers 370 million. Together with AppleBot and PerplexityBot that's close to 1.3 billion requests, a little over 28% of Googlebot's volume.

Google's own documentation says the same thing from the other side. Googlebot processes JavaScript apps in three phases (crawling, rendering, indexing) and renders with an evergreen Chromium, but pages can sit in the rendering queue for a while. The docs still recommend server-side rendering or pre-rendering, partly because "not all bots can run JavaScript".

Seeing your page the way they do

You don't need a special tool to look at your page the way these crawlers do. A crawler that doesn't run JavaScript gets the same bytes curl gets:

# The HTML as it arrives, before any script runs
curl -s https://richi.codes/article/2026-11-05-ai-crawlers-dont-run-your-javascript | grep -o '<h1[^>]*>[^<]*</h1>'
Enter fullscreen mode Exit fullscreen mode

If your headline, your text or your structured data aren't in that output, an AI crawler won't find them either.

The demo does the same thing in the browser with DOMParser. MDN describes it plainly: when you parse with text/html, <script> elements are marked as non-executable and event handlers don't run, so the resulting document is "essentially inert". That's a decent stand-in for a crawler that only reads HTML:

function readAsBot(html) {
  const doc = new DOMParser().parseFromString(html, 'text/html'); // scripts are never run
  const types = [...doc.querySelectorAll('script[type="application/ld+json"]')].map((s) => {
    try { return JSON.parse(s.textContent)['@type']; } catch { return '?'; }
  });
  doc.querySelectorAll('script, style, template').forEach((n) => n.remove());
  return {
    title: doc.title.trim(),
    description: doc.querySelector('meta[name="description"]')?.content || '',
    h1: doc.querySelector('h1')?.textContent.trim() || '',
    jsonLd: types,
    links: doc.querySelectorAll('a[href]').length,
    text: (doc.body?.textContent || '').replace(/\s+/g, ' ').trim()
  };
}
Enter fullscreen mode Exit fullscreen mode

Two details matter. The JSON-LD has to be read before the <script> elements are removed, because structured data lives inside a script tag even though it's data rather than code. And the browser pane gets its numbers from a nested sandboxed iframe that runs the same HTML for real and reports document.body.innerText back with postMessage.

Try the three presets:

  • Client-rendered SPA: an empty <div id="root"> plus a script. The browser shows a full product page. The crawler gets a generic title, no description, no <h1>, no JSON-LD, no links, and only the <noscript> message.
  • Server-rendered: the same page with everything in the HTML. Both readers agree.
  • SSR + JS widget: the main content comes from the server and only the reviews box is filled in by a script. The crawler loses exactly that box, which is fine if the reviews aren't what you want cited.

You can also open the editor and paste your own page's source (view-source, not the DOM inspector, which shows the page after JavaScript has run).

Gotchas

  • Structured data injected by JavaScript doesn't count for these bots. A tag manager that adds JSON-LD on the client gives Googlebot something and the others nothing. Vercel notes that data present in the initial HTML, JSON included, can still be picked up.
  • Hydration is fine. Server rendering plus JavaScript that takes over afterwards is exactly the "server-rendered" case: the content is already in the HTML. This site's articles are generated with Next.js getStaticProps, and the curl above returns the headline straight from the HTML.
  • Meta tags set on the client don't help either. A document.title changed in a useEffect is the "App" title the crawler sees in the first preset.
  • DOMParser is an approximation. It shows what's in the HTML; it doesn't know about robots.txt rules, redirects or how each company processes the text afterwards.

What to keep an eye on

The Vercel numbers are from late 2024, and crawlers change. If an AI company starts rendering JavaScript, it will most likely say so in its crawler documentation, so that's where to look. Until then, the safe rule is the one both Google and Vercel give: put your main content, metadata and navigation in the HTML your server sends, and keep client-side rendering for the extras.

Sources

Originally published at richi.codes.

Top comments (1)

Collapse
 
hanin_mp profile image
Hanin Mp •

Good point on JSON-LD injected through a tag manager. That is the version of this problem I see most on client sites: Googlebot renders it and the schema validates in the Rich Results Test, while the non-rendering AI crawlers get nothing. A quick extra check is to run curl with the actual crawler user agents, because a WAF or CDN rule can serve a bot a challenge page even when the raw HTML is fine for a normal request.