A web page can look complete in your browser and still send almost no useful content in its initial response.
That sounds contradictory until you separate two things developers often treat as interchangeable: the HTML returned by the server and the document produced after a browser executes the page's JavaScript.
For a person using a modern browser, the distinction is easy to miss. The browser runs the application, fetches data, updates the DOM, and presents the finished page.
A crawler does not necessarily follow the same path. Some crawlers render JavaScript. Some render it later. Some use limited execution environments. Some primarily inspect the response they receive. A crawler may also stop when a script fails, a request is blocked, or rendering exceeds its resource budget.
The useful question is therefore not just, “Does this page load?” It is, “Which version of this page is available to the system trying to read it?”
Three versions of the same page
It helps to distinguish three representations.
Initial or raw HTML is the response body returned by the server before client-side JavaScript runs. You can inspect it with an HTTP client or your browser's “View Source” feature.
Browser-rendered DOM is the document after the browser has parsed the response and executed the relevant JavaScript. This is what you normally inspect in the Elements panel of developer tools.
Crawler response is what the server returns for a particular crawler request. It may match the initial HTML sent to a normal browser, but it does not have to. A site may route known crawlers to server-rendered or prerendered HTML while sending its client application to normal visitors.
Consider a simplified client-rendered application:
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>Acme Analytics</title>
<script type="module" src="/assets/app.js"></script>
</head>
<body>
<div id="root"></div>
</body>
</html>
After JavaScript runs, the DOM might contain a product heading, several paragraphs, pricing links, documentation navigation, structured data, and customer questions. None of that information is present in the response above.
The browser has enough information to build the page. A system reading only the response does not.
CSR and hydration are related, but not identical
Client-side rendering (CSR) usually means that JavaScript creates most of the meaningful document in the browser. The server sends a shell and the client fetches or computes the content.
Hydration starts from HTML that the server has already rendered. JavaScript attaches behavior to that existing markup. A hydrated page can therefore expose substantial content before JavaScript runs—provided the server response really contains that content.
Framework choice alone does not tell you which result a crawler receives. React, Vue, Svelte, Angular, and similar tools can participate in client rendering, server rendering, static generation, or combinations of them. What matters is the deployed response for a specific URL.
This is why statements such as “React sites are invisible to crawlers” are not useful. Some React sites return complete documents. Others return a thin shell. Many fall somewhere in between.
What can disappear from the raw response?
When the initial HTML is thin, the missing material is not limited to body copy. Common gaps include:
- The primary heading and descriptive text
- Product or article content loaded through an API
- Internal navigation and contextual links
- Canonical, description, or social metadata inserted by client code
- Structured data generated after mount
- Pagination or related-content links
- Text inside tabs, accordions, or client-side routes
- Error and empty states that replace expected content when a request fails
Each gap has a different consequence. Missing body text reduces what a non-rendering reader can understand. Missing internal links affects discovery. Missing structured data removes an explicit machine-readable description, even when visible copy remains available.
A character-count difference alone cannot tell you whether a page has a serious problem. A cookie banner can add thousands of unimportant characters, while a smaller difference can contain the only product description or links to deeper pages. The comparison needs both numbers and inspection.
Search crawlers and AI crawlers are not one audience
It is tempting to divide the world into “Googlebot” and “AI bots,” but both groups contain systems with different purposes and capabilities.
A search engine may fetch a document, schedule rendering separately, revisit it with another service, and combine the result with other signals. An AI company may operate different crawlers for search indexing, model training, user-requested retrieval, and link previews. An assistant answering a live question may also use a retrieval system that behaves differently from the company's general web crawler.
Do not assume every crawler executes the same JavaScript—or that one successful render proves availability to all the others. Make the important public meaning available in useful HTML, then verify crawler-specific behavior with evidence.
This matters beyond traditional search because AI visibility has at least two separate layers:
- Can a system access and understand the page?
- Does it choose to retrieve, cite, mention, or recommend that information for a particular question?
Improving the first layer does not guarantee the second. But a page that exposes no meaningful content creates an avoidable technical obstacle.
How to compare the versions
Start with the exact public URL, not a development route or a component in isolation.
1. Fetch the initial response
Use an HTTP client and follow redirects deliberately:
curl -L https://example.com/product
Save the body and inspect it as a document. Look for the page's main heading, a distinctive sentence, important links, canonical metadata, and structured data.
Also record the final URL, status code, content type, redirect chain, and response size. A 200 response only proves that the server returned something successfully; it does not prove that the response contains the page.
2. Inspect View Source
“View Source” is a convenient representation of the server response. Do not confuse it with the Elements panel, which shows the live DOM after browser processing.
Search the source for a sentence that visibly appears on the page. If it is absent, determine whether the browser adds it during rendering.
3. Disable JavaScript
Loading the page without JavaScript is a useful diagnostic, although it is not a perfect crawler simulation. It quickly reveals whether meaningful HTML exists before the application starts.
The expected result depends on the product. A complex editor may reasonably require JavaScript to function. Its public landing page should still be understandable without waiting for an application API.
4. Render in a controlled browser
Use Playwright, Puppeteer, or another browser automation tool to capture the resulting DOM. Define what “finished” means: load may be too early, while network idle may never arrive on a page with analytics or live connections.
Compare meaningful signals rather than performing a byte-for-byte diff:
- Normalized visible text
- Headings
- Links and destinations
- Title, canonical, and description
- Robots directives
- Structured-data blocks
- Final URL and HTTP behavior
5. Repeat with relevant user agents
User-agent testing can reveal routing differences, but it must be interpreted carefully. A different response does not automatically mean that a crawler will execute or accept it. Conversely, an identical response does not show what the crawler later renders.
Check what your own infrastructure does. Avoid claiming knowledge of a crawler's internal rendering process based only on a user-agent request.
When prerendering helps
Prerendering can be useful when a public, content-oriented route sends a thin application shell and changing the application's rendering architecture is not immediately practical.
A rendering service can execute the page, capture useful HTML, cache it, and serve that representation to eligible crawlers. This can be especially practical for an established SPA where a framework migration would be disproportionate to the problem.
The output should remain current, preserve correct statuses and redirects, and represent the same public content a visitor receives. Stale or materially different content creates a new problem.
When prerendering is unnecessary
Do not add a rendering layer merely because the site uses JavaScript.
You probably do not need it when:
- Server rendering or static generation already returns the important content
- The raw and rendered versions differ only in interactive behavior
- The route is private or application-only and is not intended for discovery
- The missing content can be fixed directly with a smaller server-side change
- The page's public metadata, text, links, and structured data are already present
The best outcome is not “prerender everything.” It is “make each public page's important content reliably available with the least unnecessary complexity.”
Test the response, not the framework label
This raw-versus-rendered gap was one of the problems that led me to build Prerender Buddy. Its free checker compares the initial response, crawler-facing HTML, and browser-rendered page so the diagnosis is based on the URL's actual behavior rather than assumptions about its stack.
You do not need a product to perform the basic investigation. curl, browser source, developer tools, and a controlled browser script will take you a long way.
The important habit is to stop treating “it loads in my browser” as the end of the test. A modern page is a sequence of representations. If discovery matters, inspect the representation that arrives before the interface becomes visible.
Top comments (0)