Most SPA SEO advice answers one question: will Googlebot see my content? Googlebot fetches the HTML, queues the page, runs your JavaScript in a headless Chromium and indexes what comes out. Slow, sometimes flaky, but it gets there.
The AI crawlers don't get there. ChatGPT's crawlers fetch the HTML and stop. If your app mounts into an empty <div id="root">, that empty div is the whole page as far as ChatGPT search is concerned.
We build LovedByAI, an AI search optimization platform for small businesses. Our WordPress plugin logs every request a recognised bot makes to a site, which is where the crawler numbers below come from. This post is the developer version of our breakdown of dynamic bot rendering, a CDN feature now being pitched to marketers as the fix. It includes a standard-library Python script that shows what each crawler gets from a URL, and that spots dynamic rendering from the outside. We ran it before publishing; the output below is real.
Do AI crawlers render JavaScript?
No, not the ones that matter for ChatGPT. Vercel's December 2024 study found that none of the major AI crawlers executed JavaScript, including ChatGPT-User, OAI-SearchBot, GPTBot, ClaudeBot and PerplexityBot. ChatGPT and Claude did download JavaScript files (11.50% and 23.84% of their requests), but never ran them. The exceptions were Gemini, which uses Googlebot's rendering infrastructure, and AppleBot.
So for an SPA there are two audiences now:
-
Googlebot, which renders. Your existing SPA SEO work (rendering budget,
<title>updates, history routing, hydration errors) still applies. - Everything OpenAI, Anthropic and Perplexity send, which reads the HTML you return and nothing after it.
Which AI crawlers should an SPA care about?
AI bots come in three kinds, and they do different jobs:
| Kind | Examples | What it does | Share of AI bot requests |
|---|---|---|---|
| Training crawlers | GPTBot, ClaudeBot, Meta-ExternalAgent | Copy pages to train future models | 69.9% |
| Search index crawlers | OAI-SearchBot, PerplexityBot, Claude-SearchBot | Crawl ahead of time to build the index an assistant searches when someone asks | 18.8% |
| Live fetchers | ChatGPT-User, Claude-User, Perplexity-User | Open one page at the moment a person asks, to read it for the answer | 11.3% |
The shares are from LovedByAI's logs: the average across 924 WordPress sites running our plugin, nearly three in four of them small and mid-sized businesses, from 6 September to 5 October 2026. An app with a different audience will have a different mix, but the reach is worth knowing: in that window OAI-SearchBot crawled 90.6% of the sites and ChatGPT-User 91.7%.
The one to worry about is OAI-SearchBot. OpenAI uses it to surface websites in ChatGPT search. If it reads an empty shell, ChatGPT search has little more than your <title> to go on for that URL.
How do you see what each crawler gets from a page?
Fetch the page with each crawler's user agent and count the visible words in the raw HTML, before any JavaScript. That's what bot_view.py does. For every URL it:
- fetches as a desktop browser, Googlebot, OAI-SearchBot, ChatGPT-User and GPTBot, using the user-agent strings Google and OpenAI publish;
- reports the status, the first
<h1>and the visible word count, skipping<script>,<style>,<noscript>,<template>and<svg>; - flags an empty shell: fewer than 50 visible words, with the app root (
#root,#app,#__next,#__nuxt,#___gatsby) empty; - flags a page that differs by user agent: a different status, title, H1 or a big gap in word count compared with the browser fetch.
The second check matters more than it looks. A crawler getting a different page than a browser is what dynamic rendering, a bot-blocking firewall and cloaking all look like from outside. The script can't tell you which one it is, only that you should go and look.
#!/usr/bin/env python3
"""bot_view.py: what does each crawler get from your page before any JavaScript runs?
Fetches every URL as a desktop browser, Googlebot, OAI-SearchBot, ChatGPT-User and
GPTBot, using the user-agent strings Google and OpenAI publish, and reports the
status, the first <h1> and the number of visible words in the raw HTML (titles are
compared too).
It flags two things:
1. EMPTY SHELL: the raw HTML has almost no visible text (an app root such as
<div id="root"> with nothing inside, filled later by JavaScript). Crawlers
that don't run JavaScript, which today means OpenAI's and most other AI
crawlers, read exactly this.
2. DIFFERS BY USER AGENT: a crawler gets a different title, H1, status or
amount of text than the browser does. That's what dynamic rendering (a
pre-rendered snapshot served to some bots) or a bot-blocking firewall looks
like from the outside. Same content in a different shape is fine;
different content is cloaking.
Standard library only. Python 3.9+.
Usage: python3 bot_view.py https://example.com/ [more URLs] [--min-words 50]
Exit code 1 if anything was flagged, so it can run in CI.
"""
import argparse
import re
import sys
import urllib.error
import urllib.request
from html.parser import HTMLParser
# Published user-agent strings, October 2026. Google's docs write Googlebot's
# Chrome version as W.X.Y.Z (it follows current Chrome); any recent version works.
AGENTS = [
("Browser", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"),
("Googlebot", "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/141.0.0.0 Mobile Safari/537.36 "
"(compatible; Googlebot/2.1; +http://www.google.com/bot.html)"),
("OAI-SearchBot", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; "
"OAI-SearchBot/1.4; +https://openai.com/searchbot"),
("ChatGPT-User", "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; "
"ChatGPT-User/1.0; +https://openai.com/bot"),
("GPTBot", "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; "
"GPTBot/1.4; +https://openai.com/gptbot"),
]
# Ids that single-page apps mount into (React, Vue, Next.js, Nuxt, Gatsby, Angular-ish).
APP_ROOTS = {"root", "app", "__next", "__nuxt", "___gatsby", "svelte", "main-app"}
SKIP = {"script", "style", "noscript", "template", "svg", "head", "title"}
VOID = {"area", "base", "br", "col", "embed", "hr", "img", "input", "link", "meta",
"source", "track", "wbr"}
class PageText(HTMLParser):
"""Collects the title, first H1, visible text and any empty app root."""
def __init__(self):
super().__init__(convert_charrefs=True)
self.stack = [] # open tags, to know when we're inside <script> etc.
self.title, self.h1, self.text = [], [], []
self.in_h1 = False
self.h1_done = False
self.scripts = 0
self.root_id, self.root_depth, self.root_words = None, None, 0
self.empty_root = None
def handle_starttag(self, tag, attrs):
if tag == "script":
self.scripts += 1
if tag in VOID:
return
self.stack.append(tag)
if tag == "h1" and not self.h1_done:
self.in_h1 = True
el_id = dict(attrs).get("id")
if el_id in APP_ROOTS and self.root_id is None:
self.root_id, self.root_depth, self.root_words = el_id, len(self.stack), 0
def handle_endtag(self, tag):
if tag in VOID or tag not in self.stack:
return
while self.stack: # tolerate unclosed tags
closed = self.stack.pop()
if closed == "h1" and self.in_h1:
self.in_h1, self.h1_done = False, True
if self.root_depth is not None and len(self.stack) < self.root_depth:
if self.root_words == 0:
self.empty_root = self.root_id
self.root_depth = None
if closed == tag:
break
def handle_data(self, data):
if "title" in self.stack:
self.title.append(data)
if any(t in SKIP for t in self.stack):
return
words = data.split()
if not words:
return
self.text.extend(words)
if self.in_h1:
self.h1.append(data)
if self.root_depth is not None:
self.root_words += len(words)
def clean(parts, limit=40):
s = re.sub(r"\s+", " ", " ".join(parts)).strip()
return s if len(s) <= limit else s[: limit - 1] + "…"
def fetch(url, ua):
req = urllib.request.Request(url, headers={"User-Agent": ua, "Accept": "text/html,*/*"})
try:
with urllib.request.urlopen(req, timeout=20) as r:
status, body, ctype = r.status, r.read(5_000_000), r.headers.get_content_charset()
except urllib.error.HTTPError as e:
status, body, ctype = e.code, e.read(5_000_000), e.headers.get_content_charset()
except Exception as e: # DNS, TLS, timeout: report it as a row, keep going
return {"status": f"ERR {type(e).__name__}", "title": "", "h1": "", "words": 0,
"scripts": 0, "empty_root": None}
p = PageText()
p.feed(body.decode(ctype or "utf-8", errors="replace"))
p.close()
return {"status": status, "title": clean(p.title, 10_000), "h1": clean(p.h1, 10_000), "words": len(p.text),
"scripts": p.scripts, "empty_root": p.empty_root}
def check(url, min_words):
rows = [(name, fetch(url, ua)) for name, ua in AGENTS]
print(f"\n{url}")
print(f" {'agent':<14}{'status':<7}{'words':>6} h1")
for name, r in rows:
print(f" {name:<14}{str(r['status']):<7}{r['words']:>6,} {clean([r['h1']]) or '(no h1)'}")
flags = []
shells = {} # same reason -> the agents it applies to, so five bots make one line
for name, r in rows:
if isinstance(r["status"], int) and r["status"] < 400 and r["words"] < min_words:
why = f"{r['words']} visible words"
if r["empty_root"]:
why += f', empty <div id="{r["empty_root"]}">'
if r["scripts"]:
why += f", {r['scripts']} script tag{'s' * (r['scripts'] != 1)}"
shells.setdefault(why, []).append(name)
for why, names in shells.items():
who = "every agent" if len(names) == len(rows) else ", ".join(names)
note = " Googlebot renders JavaScript later; OpenAI's crawlers don't." if "Googlebot" in names else ""
flags.append(f"EMPTY SHELL for {who}: {why}.{note}")
base = rows[0][1]
for name, r in rows[1:]:
diffs = []
if r["status"] != base["status"]:
diffs.append(f"status {base['status']} -> {r['status']}")
if r["title"] != base["title"]:
diffs.append("title")
if r["h1"] != base["h1"]:
diffs.append("h1")
gap = abs(r["words"] - base["words"])
if gap > 50 and gap > 0.2 * max(r["words"], base["words"]):
diffs.append(f"words {base['words']:,} -> {r['words']:,}")
if diffs:
flags.append(f"DIFFERS BY USER AGENT for {name} vs Browser: {', '.join(diffs)}.")
for f in flags:
print(f" ! {f}")
if not flags:
print(" ok: every agent gets the same page, with text in the HTML.")
return bool(flags)
def main():
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
ap.add_argument("urls", nargs="+")
ap.add_argument("--min-words", type=int, default=50,
help="fewer visible words than this counts as an empty shell (default 50)")
a = ap.parse_args()
flagged = [check(u, a.min_words) for u in a.urls]
sys.exit(1 if any(flagged) else 0)
if __name__ == "__main__":
main()
What does bot_view.py show on a single-page app?
Here's the demo app we used, a plumber's homepage built the SPA way: index.html holds <div id="root"></div> and one script tag, and app.js fills in the headline, services and reviews. Served locally with python3 -m http.server 8790:
$ python3 bot_view.py http://localhost:8790/
http://localhost:8790/
agent status words h1
Browser 200 0 (no h1)
Googlebot 200 0 (no h1)
OAI-SearchBot 200 0 (no h1)
ChatGPT-User 200 0 (no h1)
GPTBot 200 0 (no h1)
! EMPTY SHELL for every agent: 0 visible words, empty <div id="root">, 1 script tag. Googlebot renders JavaScript later; OpenAI's crawlers don't.
Every agent gets the same empty shell: a <title> and no text. In a browser, the page shows a headline, three service cards and a five-star review. Googlebot gets there in its rendering pass. ChatGPT-User, OAI-SearchBot and GPTBot never do.
And here is a server-rendered page, our own GEO checklist (a Next.js page rendered on the server):
$ python3 bot_view.py https://www.lovedby.ai/blog/geo-checklist
https://www.lovedby.ai/blog/geo-checklist
agent status words h1
Browser 200 4,243 GEO Checklist: 30 On-Page Checks for AI…
Googlebot 200 4,243 GEO Checklist: 30 On-Page Checks for AI…
OAI-SearchBot 200 4,243 GEO Checklist: 30 On-Page Checks for AI…
ChatGPT-User 200 4,243 GEO Checklist: 30 On-Page Checks for AI…
GPTBot 200 4,243 GEO Checklist: 30 On-Page Checks for AI…
ok: every agent gets the same page, with text in the HTML.
Same 4,243 words for every agent, and the H1 in the HTML. That's the result you want.
What does dynamic bot rendering look like from outside?
Dynamic bot rendering is the old dynamic rendering pattern pointed at AI assistants. A rule on your CDN (CloudFront, Cloudflare or Vercel) checks the user agent for a list of assistant tokens such as chatgpt-user, claude-user and perplexity-user. Matching requests get a stored, pre-rendered snapshot of the page. Everyone else, OAI-SearchBot included, gets the live page.
To see what that looks like to the script, we built a toy version: the same demo app, plus a server that hands a pre-rendered snapshot.html to the nine assistant tokens on the published list.
ASSISTANT_TOKENS = ("chatgpt-user", "codex", "claude-user", "claude-code", "perplexity-user",
"gemini-deep-research", "duckassistbot", "amzn-user", "mistralai-user")
ROOT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "demo")
class Handler(http.server.SimpleHTTPRequestHandler):
def __init__(self, *a, **kw):
super().__init__(*a, directory=ROOT, **kw)
def do_GET(self):
ua = (self.headers.get("User-Agent") or "").lower()
if self.path in ("/", "/index.html") and any(t in ua for t in ASSISTANT_TOKENS):
self.path = "/snapshot.html"
super().do_GET()
Run against it:
$ python3 dynamic_demo.py 8793 &
$ python3 bot_view.py http://localhost:8793/
http://localhost:8793/
agent status words h1
Browser 200 0 (no h1)
Googlebot 200 0 (no h1)
OAI-SearchBot 200 0 (no h1)
ChatGPT-User 200 64 Emergency plumber in Leeds, open 24/7
GPTBot 200 0 (no h1)
! EMPTY SHELL for Browser, Googlebot, OAI-SearchBot, GPTBot: 0 visible words, empty <div id="root">, 1 script tag. Googlebot renders JavaScript later; OpenAI's crawlers don't.
! DIFFERS BY USER AGENT for ChatGPT-User vs Browser: h1, words 0 -> 64.
That's the whole trade in five rows. ChatGPT-User, the live fetcher, now reads the headline and 64 words. OAI-SearchBot, the crawler that builds ChatGPT search's index, still gets an empty <div id="root">. So does GPTBot, and so would PerplexityBot and Claude-SearchBot. A snapshot helps the fetch that happens after ChatGPT has already picked your page; it does nothing for the index it picks from.
Two more things the toy makes obvious:
- Anyone can send that user agent. The routing trusts a header. On Wednesday 7 October 2026, 69% of the 404s logged under the ChatGPT-User user agent on small business websites in LovedByAI's logs were exploit probes, not ChatGPT. If you measure a snapshot's effect by counting requests under assistant names, check them against OpenAI's published IP ranges (chatgpt-user.json, searchbot.json, gptbot.json) first.
-
A user-agent test only sees user-agent routing. If a CDN also checks the request's IP,
bot_view.pywill see the live page for every agent. That isn't proof nothing is switched.
Is it cloaking to serve ChatGPT-User a snapshot?
Not by Google's definition, while the snapshot matches the page. Google's dynamic rendering documentation says similar content isn't cloaking, and its example of cloaking is a page about cats for users and dogs for crawlers. The risk is drift: a snapshot that still shows last month's price isn't the same content as the live page any more. When bot_view.py reports DIFFERS BY USER AGENT, compare the two responses by hand.
The same page warns that "dynamic rendering was a workaround and not a long-term solution."
How do you fix SPA SEO for AI crawlers?
Put the content in the HTML you send, for every visitor. Google recommends three ways to do that:
- Server-side rendering. Next.js, Nuxt, SvelteKit, Remix and Angular's SSR all render the first response on the server, then hydrate. Every crawler gets the text; nobody gets a different page.
- Static generation (prerendering at build time). For marketing pages, docs and anything that doesn't change per request, build HTML files. Astro, Next.js static export and Nuxt's generate mode all do this. It's the cheapest fix there is: the CDN serves files.
- Hydration of server-sent HTML, the general form of both: send real markup, then let JavaScript take over.
If you can't change the app soon, serve a prerendered copy to every bot you care about, OAI-SearchBot, PerplexityBot and Claude-SearchBot included, not just the live fetchers. And give it an expiry: re-render on deploy and when prices or stock change, so the copy never drifts from the page.
Whatever you pick, put bot_view.py in CI against a staging URL. It exits with 1 when anything is flagged:
python3 bot_view.py https://staging.example.com/ https://staging.example.com/pricing
A deploy that ships an empty shell, or a CDN rule that starts sending bots a different page, then fails the build instead of quietly costing you ChatGPT search.
Where does WordPress fit?
Mostly outside this problem. WordPress builds each page in PHP and sends finished HTML, so the headings, text and schema are in the first response. The exceptions are widgets that load after the page (some review carousels, booking and menu embeds) and headless WordPress with a front end that renders in the browser.
By default, LovedByAI doesn't switch anything by user agent: people, ChatGPT-User, OAI-SearchBot and Googlebot get the same WordPress page at the same URL. Each page's machine-readable copy lives at its own public address (the page URL plus ?llm_view=1), linked from the site's footer with a plain HTML link, so any person or bot can open it.
The full breakdown of dynamic bot rendering, with the pros, the cons and how it compares with prerendering and server-side rendering: https://www.lovedby.ai/blog/dynamic-bot-rendering
For the business side, whether a small business should turn dynamic bot rendering on at all, see our analysis on Medium.
Top comments (1)
We need to produce a short YouTube comment per developer style, reference specific reaction/question about the video. No marketing. Should be casual, maybe ask about why Chrome headless vs bot, or mention script. Must comply with guidelines: start lowercase, short, one or two sentences, no quotes, no hashtags, no URLs, no punctuation issues. Avoid double hyphen, no em-dash. Use ASCII quotes only if needed. Probably something like: "noticed the python script shows different markup for ChatGPT-User, does that