The 30-Second Raw-HTML Smoke Test for AI Visibility
Your page looks perfect in the browser. Your Lighthouse score is green. Analytics look normal.
None of that tells you what an AI assistant can actually read on your page.
This came out of a genuinely good back-and-forth with @alexshev in the comments of my last post, and they framed it better than I had: treat the raw HTML check as a smoke test. It doesn't prove your site is great for AI readers. It just catches the embarrassing failures early — before you spend a month wondering why you never show up in AI answers.
Why the browser lies to you
When you load a page, your browser executes JavaScript, waits for fetches, hydrates the framework, and renders the final result. What you see is the end state.
Most AI crawlers don't do that. GPTBot, ClaudeBot, and PerplexityBot generally fetch the raw HTML document and read what's there. No rendering queue, no waiting for your useEffect to resolve. Google is the notable exception — Googlebot renders JavaScript, and AI Overviews build on that index — but assume every other AI system reads the initial response only.
So there's a gap between what humans see and what machines can reliably extract. And the gap is invisible precisely because everything looks fine.
The one thing most devs get wrong
DevTools Inspector is not raw HTML.
The Elements panel shows the live DOM — after JavaScript has run. It will happily show you content that no AI crawler will ever see.
What you want is View Source (Cmd/Ctrl + U). That's the actual document the server sent.
If your content is in the Inspector but not in View Source, it's client-rendered. That's the whole failure mode in one sentence.
The test itself
Pick your poison - all three take under a minute.
Option 1 — View Source. Cmd/Ctrl + U, then Cmd/Ctrl + F and search for your headline, your price, your address. Not there? That's your answer.
Option 2 — Disable JavaScript. In Chrome DevTools: Cmd/Ctrl + Shift + P → type "Disable JavaScript" → reload. You're now looking at roughly what a non-rendering crawler gets.
Option 3 — curl. Fastest for a quick gut check:
curl -sL https://yoursite.com | grep -i "your headline here"
Want to actually read the text a crawler would extract? This strips tags and gives you the first 2000 characters of real content:
curl -sL https://yoursite.com | python3 -c "
import sys, re
h = sys.stdin.read()
h = re.sub(r'<(script|style)[^>]>.?</\1>', ' ', h, flags=re.S)
print(re.sub(r'\s+', ' ', re.sub(r'<[^>]+>', ' ', h))[:2000])
"
If that output is mostly navigation and boilerplate — or nearly empty — you've found your problem.
The four failure modes to look for
Alex's list is the useful part here, because it's specific enough to actually run against a page:
-
Empty HTML. The classic CSR single-page app. Your is
and everything else arrives via JS. Crawlers get a shell. Missing business facts. Title, offer, location, product details. These are the things an AI needs to describe or recommend you. If they're not in the initial response, you're asking the model to guess.
Invisible offers. Pricing loaded from an API on mount. Very common, very costly — "how much does X cost" is exactly the kind of question people ask assistants.
Client-only content. Tabs, accordions, "load more," lazy-mounted sections. Content that exists but only after an interaction the crawler will never perform.
The fix for all four is the same shape: get the critical content into the initial HTML. SSR, SSG, ISR, or prerendering — whichever your stack supports. You don't have to abandon React. Next.js, Nuxt, Remix, SvelteKit all solve this by default; the problem is almost always plain CSR or content deliberately deferred to the client.
Layer two: structured data as insurance
Here's the nuance I'd add to the smoke test, because passing it isn't quite the finish line.
Even when your content is in the raw HTML, an agent can still flatten it if the structure is ambiguous. It gets the words but loses the relationships — which number is the price, which line is the address, what's the product versus a related item.
That's where JSON-LD earns its keep:
{
"@context": "https://schema.org",
"@type": "Product",
"name": "Acme Widget",
"offers": {
"@type": "Offer",
"price": "49.00",
"priceCurrency": "USD",
"availability": "https://schema.org/InStock"
}
}
That's an unambiguous, machine-readable copy of your hard facts, sitting right there in the initial HTML. It survives even a shallow skim.
So the rule of thumb ends up being two layers:
Critical content in the initial HTML — so it can be read at all.
Hard facts mirrored in structured data — so they can't be misread.
What this test does not prove
Being honest about the limits, since that's what makes a smoke test useful rather than a false comfort:
It doesn't mean your content is good, or that anyone will cite it.
It doesn't tell you whether AI crawlers are actually allowed in — that's robots.txt, a separate check entirely.
It doesn't replicate any specific product's retrieval and ranking pipeline. Nobody outside those companies can.
It catches one specific, common, expensive failure: important content hiding behind client-side timing. That's it. That's enough to be worth 30 seconds.
Run it on something today
Try it on your own site. Then try it on a competitor's — that part is genuinely educational, and you'll find broken ones faster than you'd expect.
If you'd rather not do it by hand, I maintain a set of free tools for this — no signup — that fetch server-rendered HTML the way a crawler would and score what's actually readable. But the curl one-liner above costs you nothing and catches most of it.
The cheapness is the point.
Thanks to @alexshev for the smoke-test framing and the four failure modes — this post is basically our comment thread, cleaned up.
Top comments (0)