40% of pages are empty over plain HTTP

40.6% of popular landing pages need JavaScript to show you anything useful. That is from State of Web Access, an audit Zyte (where I work) ran on 11,100 landing pages. The test was blunt. Fetch the page over plain HTTP, fetch it again in a headless browser, and if rendering grew the meaningful HTML by more than half, count it as JavaScript-dependent.

On the pages that failed the test, rendering added 140,923 bytes on average. Three quarters of the final HTML did not exist until a browser ran the scripts. Restaurants and airlines top the list at 66%, travel at 65%.

The report calls this architecture rather than defence, and I agree. Single page apps won. The HTML that comes over the wire is a shell and the content arrives in a second round trip.

What rendering costs you

The instinct on seeing an empty response is to reach for Playwright. Here is what that does to the bill, using the audit's own tier data. Zyte's tiers map to the infrastructure needed to fetch a page reliably, and the report scored every site both ways:

Access method	Simple	Easy	Moderate	Complex	Advanced	Mean tier
Plain HTTP	85.2%	9.4%	2.5%	2.0%	0.9%	1.24
Headless browser	58.4%	31.3%	8.0%	1.8%	0.5%	1.55

The gap is not sites defending themselves. A browser carries a heavier fingerprint and a heavier compute cost even on a site with no barriers at all. Switching to rendering moves a quarter of all sites up a tier for no reason other than the method.

So before rendering, I look in three places. The data is very often already in the response.

First, measure

This is the audit's test, rewritten to compare visible text instead of HTML bytes, which is stricter. Run it once per website before you decide anything.

import asyncio
import sys

import httpx
from playwright.async_api import async_playwright
from selectolax.parser import HTMLParser

UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"


def visible_text_len(html: str) -> int:
    tree = HTMLParser(html)
    for node in tree.css("script, style, noscript, template"):
        node.decompose()
    return len(tree.body.text(separator=" ", strip=True)) if tree.body else 0


async def js_dependence(url: str) -> dict:
    plain = httpx.get(url, headers={"User-Agent": UA}, follow_redirects=True, timeout=20).text
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page(user_agent=UA)
        await page.goto(url, wait_until="networkidle")
        rendered = await page.content()
        await browser.close()
    a, b = visible_text_len(plain), visible_text_len(rendered)
    return {"plain_chars": a, "rendered_chars": b, "js_dependent": b > a * 1.5}


if __name__ == "__main__":
    print(asyncio.run(js_dependence(sys.argv[1])))

If js_dependent is false, stop here. You have a plain HTTP site and a browser will only make it slower and more expensive.

Second, harvest the state the framework left behind

SPAs hydrate from data the server already embedded. Next.js puts it in a script tag with the id __NEXT_DATA__. Nuxt uses a global. Apollo, Redux and a dozen others assign to window.__SOMETHING__. E-commerce and news sites also carry JSON-LD for search engines, which is often cleaner than the visible page.

A Scrapy spider that pulls all of it from the plain response, with no browser involved:

import json
import re

import scrapy

STATE_SELECTORS = {
    "next": "script#__NEXT_DATA__::text",
    "jsonld": 'script[type="application/ld+json"]::text',
}

STATE_GLOBALS = [
    r"window\.__INITIAL_STATE__\s*=\s*",
    r"window\.__PRELOADED_STATE__\s*=\s*",
    r"window\.__APOLLO_STATE__\s*=\s*",
]


class EmbeddedStateSpider(scrapy.Spider):
    name = "embedded_state"

    def parse(self, response):
        for source, sel in STATE_SELECTORS.items():
            for raw in response.css(sel).getall():
                try:
                    yield {"source": source, "url": response.url, "data": json.loads(raw)}
                except json.JSONDecodeError:
                    self.logger.debug("unparseable %s block on %s", source, response.url)

        for script in response.css("script::text").getall():
            for pattern in STATE_GLOBALS:
                m = re.search(pattern, script)
                if m:
                    obj = self._leading_json(script[m.end():])
                    if obj is not None:
                        yield {"source": "global", "url": response.url, "data": obj}

    @staticmethod
    def _leading_json(s: str):
        # Parses the JSON object at the start of s and ignores whatever follows
        # (a semicolon, more script). Avoids trying to find the closing brace by hand.
        try:
            obj, _ = json.JSONDecoder().raw_decode(s)
            return obj
        except json.JSONDecodeError:
            return None

The raw_decode trick is the useful part. You never have to find the end of the object yourself. Nuxt is the awkward one, since its global is usually a function call rather than a literal, and you either evaluate it or fall through to the next step.

Third, replay the request the page makes

Open DevTools, Network tab, filter to Fetch/XHR, reload. The JSON endpoint the page calls is right there, and it usually returns cleaner data than the HTML ever will. Right click, copy as cURL, and check what it needs. If it works with a static header set, you have a plain HTTP scraper for a JavaScript site. If it needs a token that a script computes on the client, that is your signal to render, and now you know why.

When rendering is the right call

Three cases. The API is signed or needs a challenge token from a script. The content genuinely does not exist until a user interacts. Or the TLS probe and the 403 classifier from the earlier posts in this series put the site at Moderate or above, at which point you need a browser-shaped session anyway and rendering comes with it.

Everything else is a plain HTTP job wearing a React costume.

The full JavaScript breakdown by industry and site size is on the report page, and the tier tables are under scraping cost. The tier model is Zyte's own pricing model applied to Zyte's own data, so trust the ordering between industries more than the absolute percentages.

Disclosure: I work at Zyte. The audit is ours. The code and the opinions are mine.