Scraping JavaScript-Heavy SPAs: Render Timing, Wait Strategies, and When to Reverse the API

Every scraper that hits a modern SPA eventually goes through the same five stages of grief. It starts with requests and an empty <div id="root">. Then someone says "just use Playwright." Then someone else adds time.sleep(5) and it mostly works, until the day it doesn't, and you're paging through logs trying to figure out why 30% of your jobs returned a page with no data on it.

I've built enough of these pipelines now to believe the problem usually isn't the tools — it's that nobody stopped to ask the one question that actually determines the architecture: where does the data live? For a server-rendered page, the answer is the HTML. For a JavaScript-heavy SPA, the data usually lives in a JSON payload that the frontend fetches over XHR and then paints into the DOM. That means you have two places to read it: after the paint (DOM scraping with a browser) or at the source (hitting the site's own JSON endpoints). Most teams default to full-browser rendering for everything when, in my experience, roughly half of the pages on a typical SPA target could be fetched from the underlying API at 10x less cost. This post is about how to figure out which half you're in, and how to do both sides correctly.

Why time.sleep() After Load Is Fragile

The instinctive fix for SPA timing is a fixed sleep after page.goto():

page.goto(url)
time.sleep(5)  # "give the JS time to finish"

This fails in both directions. On a fast page, you burn 5 seconds per URL doing nothing. On a slow page — cold server, throttled proxy, a heavy bundle, a slow third-party analytics script — 5 seconds isn't enough and you scrape an empty shell. The root problem: SPA render time is not a constant, it's a distribution, and it shifts with the site's deploys, your proxy latency, and the time of day. Any fixed number is a guess that's wrong for some fraction of traffic, and that fraction becomes your silent data-quality loss rate. You can't alert on it, because nothing errors — you just store empty strings.

The fix is to wait for signals, not for time. Playwright gives you three levels, and it matters which one you pick.

networkidle vs domcontentloaded vs Selector Waits

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    # Level 1: fires when HTML is parsed, before JS finishes
    page.goto(url, wait_until="domcontentloaded")

    # Level 2: fires when no network requests for 500ms
    # page.goto(url, wait_until="networkidle")

    # Level 3: wait for the actual thing you need
    page.wait_for_selector("[data-testid='product-card']", timeout=15_000)
    browser.close()

Strategy	What it means	When it's right	Failure mode
domcontentloaded	HTML parsed, scripts still running	Navigation; you'll wait explicitly afterwards	Almost never sufficient alone for SPAs
networkidle	No network activity for 500ms	Simple pages with finite requests	Never fires on pages with polling, analytics beacons, WebSockets, or ads — I've seen networkidle hang forever on dashboards
wait_for_selector	A specific element exists in DOM	Almost always the right default	You need a stable, data-adjacent selector
wait_for_function	Arbitrary JS predicate is true	Custom "data is loaded" checks	Slightly more code

My default pattern is goto(..., wait_until="domcontentloaded") followed by an explicit wait for the element that implies the data arrived — not the loading spinner disappearing (spinners hide before data paints in some frameworks), but the actual data node. If the site's data-adjacent selectors are unstable, wait_for_function on a check like window.__APP_STATE__ && window.__APP_STATE__.loaded (or checking document.querySelectorAll('.card').length > 0) is more robust.

Waiting for Hydration

There's a subtler trap with modern frameworks (Next.js, Nuxt, Remix): the HTML looks complete — the selector exists — because the server shipped a full render, but the page isn't interactive yet. Clicking a button before hydration silently does nothing. If you need interactions, wait for hydration, not just for DOM presence. A practical check:

page.goto(url, wait_until="domcontentloaded")
page.wait_for_selector("#app")  # markup exists
# Hydration: React attaches listeners; a common tell is a data attribute
# or flag the app sets once it's interactive.
page.wait_for_function(
    "() => document.querySelector('#app')?.__reactContainer$ !== undefined"
    " || window.__hydrated === true",
    timeout=10_000,
)

There's no universal hydration flag, so inspect your target once in devtools and encode whatever tell it exposes. The alternative that works everywhere is a MutationObserver — wait until the DOM stops churning:

page.goto(url, wait_until="domcontentloaded")
page.wait_for_function("""
    () => new Promise(resolve => {
        let timer;
        const obs = new MutationObserver(() => {
            clearTimeout(timer);
            // resolve after 800ms of zero DOM mutations
            timer = setTimeout(() => { obs.disconnect(); resolve(true); }, 800);
        });
        obs.observe(document.body, {childList: true, subtree: true});
        timer = setTimeout(() => { obs.disconnect(); resolve(true); }, 800);
    })
""", timeout=20_000)

This is effectively "networkidle for the DOM," and it's the single most reliable generic wait I've found for render timing — it self-adjusts to fast and slow pages.

Client-Side Routing: URL Changes Without Reloads

SPAs don't reload on navigation — they swap route components and fire fresh XHRs. Two consequences: your goto()-based habits break, and page.on("response") listeners become your best friend for capturing data mid-session. When clicking through a list-detail flow, don't goto() each detail URL (that's a full reload each time — slow and fingerprint-suspicious). Click, then wait on the route:

with page.expect_response(lambda r: "/api/items/" in r.url, timeout=10_000):
    page.click("a.card")
page.wait_for_url("**/items/*")  # SPA route change, no reload

Reverse the API: Listen to page.on("response")

Here's the thesis in practice. While your browser renders the page, it fetches exactly the JSON you're trying to scrape out of the DOM. So before writing a single DOM selector, spend ten minutes watching the network traffic:

from playwright.sync_api import sync_playwright
import json

def discover_endpoints(url: str) -> list[dict]:
    hits = []
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()

        def on_response(resp):
            ctype = resp.headers.get("content-type", "")
            if "application/json" in ctype and resp.request.resource_type in ("xhr", "fetch"):
                try:
                    body = resp.json()
                except Exception:
                    return
                hits.append({
                    "url": resp.url,
                    "status": resp.status,
                    "method": resp.request.method,
                    "payload": body if len(json.dumps(body)) < 50_000 else "...",
                })

        page.on("response", on_response)
        page.goto(url, wait_until="networkidle")
        page.wait_for_timeout(2000)   # give client-side fetches time to fire
        browser.close()
    return hits

for h in discover_endpoints("https://some-spa-site.com/products?category=shoes"):
    print(h["method"], h["status"], h["url"])

Run this on a target and you'll typically find something like /api/v2/catalog?category=shoes&page=1 returning clean, structured JSON — every field you were about to scrape out of divs, plus fields the UI never renders (full SKUs, internal IDs, pagination totals). Once you know the endpoint, you can often drop the browser entirely:

import httpx

HEADERS = {
    "accept": "application/json",
    "x-requested-with": "XMLHttpRequest",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...",
}

def fetch_catalog_page(page_num: int, proxy: str | None = None) -> dict:
    with httpx.Client(proxy=proxy, headers=HEADERS, timeout=15) as client:
        r = client.get("https://some-spa-site.com/api/v2/catalog",
                       params={"category": "shoes", "page": page_num})
        r.raise_for_status()
        return r.json()

Watch for the usual gotchas: auth tokens minted by the page JS (you may need one browser run to harvest headers/tokens, then pure HTTP after), request signing (some endpoints require headers computed client-side), and cookie/session binding. And re-verify periodically — internal APIs change without notice, so keep a canary check that alerts when the response shape drifts.

The Cost Math

Be honest about the tradeoff before committing. Rough numbers from workloads I run:

Dimension	Full-browser rendering	Direct JSON endpoint
Requests per minute per worker	~5–15	100+
CPU/RAM per request	A Chromium tab	Negligible
Bandwidth per page	1–5 MB (bundle, fonts, images)	20–200 KB
Fragile to UI redesigns	Yes — selectors break	No (until API changes)
Fragile to API changes	No	Yes
Handles anti-bot JS challenges	Yes	Only if unauthenticated
Setup cost	Low	Medium (reverse-engineering)

The pattern that wins is hybrid: one Playwright pass (with a residential proxy so the exit IP looks like a real user, and to reach geo-variant responses) to discover endpoints and harvest session tokens, then httpx at volume against the JSON API, with the browser reserved for the endpoints that turn out to be token-gated or challenge-protected. On one catalog-heavy target, this took a crawl from ~40 pages/minute on a rack of browser workers to 400+/minute on a single HTTP client process — and data quality went up, because JSON parsing doesn't mis-split a badly spaced price string the way DOM text extraction does.

Hybrid in Practice: One Browser Run, Then a Thousand HTTP Requests

The hybrid pattern deserves a concrete shape, because it's where most of the savings live. The idea: use the browser exactly once per session to establish identity — run the site's own JavaScript, let it mint whatever cookies and tokens it wants, harvest the headers the frontend sends with its API calls — then replay that identity from a plain HTTP client for the next few hundred requests.

from playwright.sync_api import sync_playwright
import httpx

PROXY = {"server": "http://p.thordata.com:9000",
         "username": "thor_user-session-h1-cc-us", "password": "thor_pass"}

def harvest_session(target: str) -> dict:
    """One browser pass: capture cookies + the headers the app's XHRs use."""
    captured_headers = {}
    with sync_playwright() as p:
        browser = p.chromium.launch(proxy=PROXY, headless=True)
        page = browser.new_page()

        def on_request(req):
            if "/api/" in req.url and req.resource_type in ("xhr", "fetch"):
                captured_headers.update(req.headers)

        page.on("request", on_request)
        page.goto(target, wait_until="networkidle")
        cookies = {c["name"]: c["value"] for c in page.context.cookies()}
        browser.close()
    return {"headers": captured_headers, "cookies": cookies}

# Then: hundreds of cheap httpx calls against the JSON API,
# rotating to a fresh harvested session every N requests or on 401/403.

Two operational notes from running this at volume. First, rotate harvested sessions proactively — most internal APIs have no per-request bot detection but do have rate limits and session-lifetime limits, and one exit IP making 500 sequential API calls looks exactly like what it is. Binding each harvested session to a sticky proxy session (Thordata's username-suffix sessions, for instance) keeps the cookie jar and the IP in lockstep, which is the same session-alignment principle I covered in my post on Playwright proxy configuration. Second, always keep a browser-based fallback path in the worker: when the API suddenly starts returning 403s or a different response shape, escalate that URL to full rendering instead of failing the job. The discovery sniffer from the previous section doubles as the diagnostic tool — run it when response shapes drift and you'll usually find the API moved or grew a new required header.

Decision Checklist

Before you write the next Playwright selector:

1. Open devtools' network tab (or run the page.on("response") sniffer above). Does a clean JSON endpoint deliver the data? If yes → hit it directly.
2. If the endpoint is token-gated: can one browser run mint a token that works for N requests? Hybrid approach.
3. Only if the data genuinely exists only post-render (canvas charts, WebGL, logged-in client-computed values): full-browser, domcontentloaded + MutationObserver wait, explicit selector waits, and client-side-route-aware navigation.
4. Never time.sleep() as a correctness mechanism — only as jitter for behavior realism.

The highest-leverage decision in SPA scraping isn't which browser or which wait strategy — it's choosing where the data lives. Get that right and most of your fleet stays cheap, fast, HTTP-based; get it wrong and you're paying Chromium prices for JSON you could have fetched directly.

Disclosure: I use Thordata's residential proxies for the geo-distributed rendering and API-reverse-engineering workloads described in this post. If you want to try them, they're at thordata.com, and the code **thor020* gets you 10% off.*