DEV Community

Cover image for Stop Scraping DOMs When JSON APIs Are Hiding in Plain Sight: A Small Benchmark with Async Python & Playwright
Trung Thanh Tran
Trung Thanh Tran

Posted on Edited on

Stop Scraping DOMs When JSON APIs Are Hiding in Plain Sight: A Small Benchmark with Async Python & Playwright

A lot of scraping code I come across (in tutorials, forum questions and job posts) runs a full headless Chromium just to extract product prices, real estate listings, or B2B directory profiles.

The pattern usually looks like this:

  1. Launch a full browser context per worker (often hundreds of MB of RAM each).
  2. Download megabytes of JavaScript bundles, web fonts, and hero images.
  3. Wait seconds for React or Vue to hydrate a JSON payload into <div> nodes.
  4. Run brittle CSS selectors (.card_wrapper__x92a > span:nth-child(2)) to turn those <div> nodes back into JSON.

Think about step 3 and step 4 together. The server already serialized clean, typed JSON from its database. The browser burned CPU cycles turning that JSON into HTML, and your scraper burned even more CPU cycles reverse-engineering HTML back into JSON—while breaking every time a frontend engineer updates a Tailwind or CSS-Modules hash.

In my view, headless browsers work best as your network-discovery tool and auth-handoff bridge, not your primary HTML parser.

Here is the 4-Tier Extraction Waterfall I use in my own scrapers and Apify Actors, with Python 3.11 code and one small, reproducible benchmark.


What One Small Benchmark Shows

To get a rough feel for the gap, my web-scraping-showcase repo has a compare-scroll script that extracts the same 100 quotes from quotes.toscrape.com (a public sandbox built for scraper testing) in two ways: Playwright scrolling the infinite-scroll page, and plain httpx calling the JSON endpoint behind it (/api/quotes?page=N, 10 pages).

Extraction Strategy Total Time (100 items, single run) Memory Selector Breakage Risk
Playwright headless Chromium (scroll + CSS selectors) ~4.80 s ~180 MB (rough estimate, not measured) High (UI class renames)
Direct JSON API via httpx ~1.45 s < 15 MB (rough estimate, not measured) Low (structured JSON)

That run came out about 3.3x faster for the JSON API. Treat it as an illustration, not a law: it is one small site, the API loop fetches pages one at a time, the Playwright time includes browser startup, and when I re-ran it on 2 Nov 2026 the ratio ranged from about 1.6x to 7.4x, mostly depending on network latency and Chromium cold start. The memory figures are rough estimates printed by the script, not measurements. The direction is consistent, though: no browser means less to download, less RAM and fewer moving parts, and a small container is usually enough for this kind of job.


The 4-Tier Extraction Waterfall

Before writing a single page.locator() or BeautifulSoup CSS query, inspect every target domain through four progressively heavier tiers:

Tier 1: application/ld+json (Schema.org) & Sitemaps

Any site that cares about Google Search rich snippets—Shopify stores, WooCommerce catalogs, healthcare directories, hospitality platforms, job boards—embeds machine-readable Schema.org entities right inside <script type="application/ld+json">.

Unlike visual DOM classes, JSON-LD tends to stay stable, because site owners have a reason to keep it working: Google reads it for rich results.

import json
from typing import Any, Iterator
from bs4 import BeautifulSoup


def iter_json_ld_entities(html: str) -> Iterator[dict[str, Any]]:
    """Extracts flattened Schema.org entities, handling nested @graph arrays."""
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup.find_all("script", attrs={"type": "application/ld+json"}):
        raw = (tag.string or tag.get_text() or "").strip()
        if not raw:
            continue
        try:
            payload = json.loads(raw)
        except json.JSONDecodeError:
            continue

        stack = payload if isinstance(payload, list) else [payload]
        while stack:
            node = stack.pop()
            if not isinstance(node, dict):
                continue
            if isinstance(node.get("@graph"), list):
                stack.extend(node["@graph"])
            if "@type" in node:
                yield node
Enter fullscreen mode Exit fullscreen mode

On many Shopify stores, combining the public /products.json?limit=250&page=1 endpoint with Product JSON-LD can give you SKUs, availability, prices and compare-at prices (plus currency and GTIN/MPN when the theme includes them) without executing a single line of client JavaScript. (My E-Commerce Price & Stock Monitor Actor currently implements only the JSON-LD half: it reads Product JSON-LD first and falls back to common HTML product cards. It does not call /products.json yet.)


Tier 2: SSR Hydration Payloads (__NEXT_DATA__, __NUXT__, RSC Flight)

Frameworks like Next.js, Nuxt, and Remix fetch data on the server to render SEO-friendly HTML, then serialize the exact server props into the document so client-side React doesn't have to refetch on mount.

  • Next.js Pages Router: Look for <script id="__NEXT_DATA__" type="application/json">. Even better, once you grab the buildId from __NEXT_DATA__, subsequent page navigations are pure JSON endpoints at /_next/data/<buildId>/catalog.json?page=2!
  • Next.js App Router (React Server Components): Inspect self.__next_f.push([1, "..."]) chunks or send the RSC: 1 header to inspect Flight stream data.

Tier 3: Automated XHR / GraphQL API Discovery with Playwright

What if the initial HTML shell is empty and data loads dynamically on scroll or filter clicks? Instead of manually clicking through Chrome DevTools for 20 minutes, run a small Playwright XHR Sniffer once during development to map every JSON endpoint the page calls:

import asyncio
from playwright.async_api import async_playwright, Response


async def sniff_json_endpoints(target_url: str) -> None:
    """Captures background JSON XHR/Fetch endpoints, status codes, and payload keys."""
    discovered: list[dict] = []

    async def on_response(response: Response) -> None:
        ctype = response.headers.get("content-type", "").lower()
        if response.request.resource_type in ("xhr", "fetch") and "json" in ctype:
            try:
                body = await response.json()
                top_keys = list(body.keys())[:6] if isinstance(body, dict) else f"list[{len(body)}]"
                discovered.append({
                    "method": response.request.method,
                    "url": response.url,
                    "status": response.status,
                    "keys": top_keys,
                    "auth_headers": {
                        k: v for k, v in response.request.headers.items()
                        if k.lower() in ("authorization", "x-api-key", "x-csrf-token")
                    },
                })
            except Exception:
                pass

    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        page.on("response", on_response)

        # Abort heavy visual assets so discovery finishes faster
        await page.route(
            "**/*.{png,jpg,jpeg,webp,svg,woff,woff2,mp4}",
            lambda route: route.abort(),
        )
        await page.goto(target_url, wait_until="networkidle")
        await browser.close()

    for hit in discovered:
        print(f"[{hit['status']}] {hit['method']} {hit['url']}\n    Keys: {hit['keys']} | Auth: {hit['auth_headers']}")


if __name__ == "__main__":
    asyncio.run(sniff_json_endpoints("https://example-directory.com/search"))
Enter fullscreen mode Exit fullscreen mode

For example, when I built a personal-project Apify Actor for Malaysia's KPKT TEDUH housing-developer portal (apify-malaysia-kpkt-teduh-property-scraper), the page turned out to load its data from a clean backend JSON route, /api/developer-swasta?page=1, which returns 25 developers per page (as of November 2026) with their projects nested inside. That is much easier to work with than the rendered table.


Tier 4: Async httpx Client with Backoff & Hybrid Cookie Handoff

Once you know the JSON endpoint, don't hammer it with naive asyncio.gather(*[client.get(u) for u in thousand_urls]). Unbounded concurrency can trigger WAF rate limits (429 Too Many Requests), exhaust ephemeral TCP sockets, and get your IP banned.

A resilient async extractor needs three guardrails:

  1. Bounded concurrency via asyncio.Semaphore.
  2. Connection pooling via a single shared httpx.AsyncClient (add http2=True if you install httpx[http2]).
  3. Exponential backoff with full jitter respecting Retry-After headers.
import asyncio
import random
from typing import Any
import httpx


class ResilientJSONExtractor:
    def __init__(
        self,
        concurrency: int = 8,
        max_retries: int = 4,
        cookies: dict[str, str] | None = None,
        extra_headers: dict[str, str] | None = None,
    ) -> None:
        self.sem = asyncio.Semaphore(concurrency)
        self.max_retries = max_retries
        self.headers = {
            "User-Agent": (
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
            ),
            "Accept": "application/json, text/plain, */*",
            **(extra_headers or {}),
        }
        self.cookies = cookies or {}

    async def fetch_page(self, client: httpx.AsyncClient, url: str, params: dict[str, Any]) -> dict[str, Any]:
        async with self.sem:
            for attempt in range(1, self.max_retries + 1):
                try:
                    resp = await client.get(url, params=params)
                    if resp.status_code == 200:
                        return resp.json()

                    if resp.status_code in (429, 502, 503, 504):
                        retry_after = float(resp.headers.get("Retry-After", 0))
                        backoff = max(retry_after, (2 ** (attempt - 1)) + random.uniform(0.1, 0.8))
                        await asyncio.sleep(backoff)
                        continue

                    resp.raise_for_status()
                except (httpx.TransportError, httpx.TimeoutException):
                    if attempt == self.max_retries:
                        raise
                    await asyncio.sleep((2 ** (attempt - 1)) + random.uniform(0.1, 0.6))
            raise RuntimeError(f"Exhausted {self.max_retries} retries for {url}")

    async def paginate_cursor(self, base_url: str, max_pages: int = 50) -> list[dict[str, Any]]:
        limits = httpx.Limits(max_keepalive_connections=20, max_connections=30)
        records: list[dict[str, Any]] = []
        async with httpx.AsyncClient(
            headers=self.headers,
            cookies=self.cookies,
            timeout=httpx.Timeout(15.0, connect=5.0),
            limits=limits,
            follow_redirects=True,
        ) as client:
            tasks = [self.fetch_page(client, base_url, {"page": p, "limit": 50}) for p in range(1, max_pages + 1)]
            pages = await asyncio.gather(*tasks, return_exceptions=True)
            for payload in pages:
                if isinstance(payload, dict) and isinstance(payload.get("data"), list):
                    records.extend(payload["data"])
        return records
Enter fullscreen mode Exit fullscreen mode

One caveat: Retry-After can also be an HTTP date rather than a number of seconds, and float() will raise on that. The shared client in my web-scraping-showcase repo handles both forms.


When You Do Need Playwright: The Hybrid Session Handoff Pattern

Sometimes pure httpx hits a wall on request #1:

  • Short-lived signed session cookies minted by a client-side JavaScript challenge.
  • OAuth / SSO login flows with dynamic CSRF tokens or MFA steps.
  • GraphQL persistedQuery hashes accompanied by rotating x-client-transaction-id headers.

Even then, you still don't need Playwright to render 10,000 detail pages! Use Hybrid Session Handoff:

  1. Launch Playwright once at job startup (a few seconds).
  2. Complete authentication or let the challenge cookie settle.
  3. Export context.cookies() into a dict ({c["name"]: c["value"] for c in await context.cookies()}) and shut down Chromium immediately.
  4. Feed those cookies into ResilientJSONExtractor and run the remaining 9,999 requests as plain JSON calls, with no browser in memory.

If cookies expire every 20 minutes, wrap the handoff in an async token-refresh lock that only re-launches Playwright on a 401/403 status code.


Putting It Into Practice (Public Example Repos)

If you want to inspect or fork small example projects built around this idea, check out these repositories:

  • web-scraping-showcase: Small async Python examples: the Playwright-vs-JSON-API benchmark above, a Playwright XHR sniffer, and CSV/JSON/SQLite exporters.
  • apify-ecommerce-price-and-stock-monitor: Async extractor that reads Schema.org Product JSON-LD first and falls back to common HTML product cards (published on Apify Store; still new, with only a few runs so far).

Next time someone hands you a slow, crashing Selenium or Puppeteer script, pop open the Fetch/XHR tab first. Surprisingly often, the backend API you wish existed is already right there waiting for you.


I'm Trung Thanh Tran, a Python developer building small web scrapers, Apify Actors and automation tools.

  • Want 6 ready-to-run Python 3.11 CLI engines (including a 4-Tier Universal Web & JSON Extractor) + 3 importable n8n/Make blueprints? Grab my Python, AI & n8n Scraper Toolkit Pro on Gumroad.
  • Need a custom scraper, anti-bot audit, or end-to-end automation pipeline? Reach out on Upwork or Fiverr.

Top comments (2)

Collapse
 
lucas_him_f7e05624dd9cc96 profile image
Lucas Him •

In paginate_cursor, gather(return_exceptions=True) lets a failed page disappear from the output while successful pages still look like a complete catalog. Could you return failed page IDs and a completeness flag alongside records, and keep the previous price-monitoring baseline whenever any page fails? Otherwise a transient timeout can look like products were removed.

Collapse
 
tthanh2k6 profile image
Trung Thanh Tran •

Spot-on catch, Lucas — silent partial harvests are the #1 cause of false "removed/delisted SKU" alerts in diff-based price and stock monitors.

In the benchmark snippet I kept paginate_cursor minimal to isolate transport overhead, warranted only for stateless one-off exports. In a stateful price-monitoring pipeline, you never want return_exceptions=True to silently drop failed pages into the same bucket as a clean run.

Here is the exact pattern I use in production to return explicit failed_page_ids + is_complete, run a targeted retry pass on stragglers, and gate baseline pruning downstream:

from dataclasses import dataclass

@dataclass(slots=True)
class HarvestResult:
    records: list[dict[str, Any]]
    failed_pages: list[int]

    @property
    def is_complete(self) -> bool:
        return len(self.failed_pages) == 0


async def paginate_cursor_safe(self, base_url: str, max_pages: int = 50) -> HarvestResult:
    limits = httpx.Limits(max_keepalive_connections=20, max_connections=30)
    records: list[dict[str, Any]] = []
    failed_pages: list[int] = []

    async with httpx.AsyncClient(
        headers=self.headers,
        cookies=self.cookies,
        timeout=httpx.Timeout(15.0, connect=5.0),
        limits=limits,
        follow_redirects=True,
    ) as client:
        page_nums = list(range(1, max_pages + 1))
        results = await asyncio.gather(
            *(self.fetch_page(client, base_url, {"page": p, "limit": 50}) for p in page_nums),
            return_exceptions=True,
        )

        for page_no, payload in zip(page_nums, results):
            if isinstance(payload, Exception) or not isinstance(payload, dict):
                failed_pages.append(page_no)
                continue
            data = payload.get("data")
            if isinstance(data, list):
                records.extend(data)
            else:
                failed_pages.append(page_no)

    return HarvestResult(records=records, failed_pages=failed_pages)
Enter fullscreen mode Exit fullscreen mode

Then in the persistence layer, we split upserts from removals:

  1. Always safe (upsert): Update price / in_stock / last_seen_at for every SKU present in result.records, even on a partial run, so live price drops still trigger alerts immediately.
  2. Gated on result.is_complete (prune / mark_removed): Only diff against the previous baseline to mark unseen SKUs as delisted when result.is_complete is True (and total count doesn't trip a sanity-drop circuit breaker, e.g., len(records) < 0.8 * prev_count). If any page failed, we retain the previous baseline for missing SKUs and queue failed_pages for a narrow retry.

Really appreciate you calling this out — I'll add a note right below that snippet so readers building stateful monitors don't trip on silent partial gathers!