Quick test before you read on. Run this against one of your own content pages:
curl -s https://yoursite.com/some-page | grep -i "a sentence from...
For further actions, you may consider blocking this person and/or reporting abuse
This is becoming a real web quality check. If the important content only exists after client-side work, humans may still see the page while crawlers, agents, and answer engines see a thin shell. The 30-second check is useful because it makes that failure visible fast.
Exactly — that's the gap I kept running into. The page looks complete in a browser because the JS runs, so it's easy to assume everything's fine. But GPTBot, ClaudeBot, and most answer-engine fetchers don't execute JavaScript the way a full browser does (or they do it inconsistently and on their own budget), so they often just get the initial HTML shell.
The tricky part is it's invisible until you look for it — your analytics, your Lighthouse score, your own eyes all say the page is fine. That's why I leaned on the "fetch the raw HTML and show what's actually there" approach: it makes the shell-vs-rendered gap obvious in one glance instead of you having to reason about hydration.
The fix usually isn't "abandon React" either — SSR/SSG or prerendering the critical content gets you most of the way there. The check is really just about knowing which bucket you're in.
That raw-HTML check is the right instinct because it removes the comforting browser illusion. If the first response is mostly a shell, then the page is asking crawlers and answer engines to do extra work before they can even understand the business.
The practical test I like is simple: if the title, offer, location, product facts, and primary content are not visible in the initial HTML, assume some important agents will miss or flatten them.
Well put, Alex — that checklist (title, offer, location, product facts, primary content) is basically a "does the raw HTML stand on its own?" test, and I like that it's concrete enough to actually run.
The one thing I'd add: it's not always binary. Even when content is in the initial HTML, agents can still flatten it if the structure is ambiguous — they get the words but lose the relationships (which number is the price, which line is the location, what's the product vs. a related item). That's where JSON-LD earns its keep. Putting your key facts — Product, LocalBusiness, Offer, price, address — into structured data in the raw HTML gives agents an unambiguous, machine-readable copy that survives even if they only skim the rendered text. So my rule of thumb ends up being two layers: get the critical content into the initial HTML and mirror the hard facts in schema, so there's no way for an agent to guess wrong.
Curious whether you lean on structured data for this, or prefer to keep the HTML itself clean enough that it isn't needed?
That is the practical line for me too. The rendered page can look perfect while the raw document is almost empty, and that creates a gap between what humans see and what crawlers or answer engines can reliably extract.
I would treat the raw HTML check like a smoke test. It does not prove the site is great, but it quickly reveals when the most important business facts are hiding behind client-side timing.
"Smoke test" is the perfect way to frame it, Alex — cheap to run, doesn't prove excellence, but instantly exposes the failure mode that matters. And you nailed the core problem in one line: the gap between what humans see and what machines can reliably extract. That gap is invisible precisely because everything looks fine.
Good conversation - this is the kind of thing more teams should be checking before they worry about anything fancier.