A lot of scraping code I come across (in tutorials, forum questions and job posts) runs a full headless Chromium just to extract product prices, real estate listings, or B2B directory profiles.
The pattern usually looks like this:
- Launch a full browser context per worker (often hundreds of MB of RAM each).
- Download megabytes of JavaScript bundles, web fonts, and hero images.
- Wait seconds for React or Vue to hydrate a JSON payload into
<div>nodes. - Run brittle CSS selectors (
.card_wrapper__x92a > span:nth-child(2)) to turn those<div>nodes back into JSON.
Think about step 3 and step 4 together. The server already serialized clean, typed JSON from its database. The browser burned CPU cycles turning that JSON into HTML, and your scraper burned even more CPU cycles reverse-engineering HTML back into JSON—while breaking every time a frontend engineer updates a Tailwind or CSS-Modules hash.
In my view, headless browsers work best as your network-discovery tool and auth-handoff bridge, not your primary HTML parser.
Here is the 4-Tier Extraction Waterfall I use in my own scrapers and Apify Actors, with Python 3.11 code and one small, reproducible benchmark.
What One Small Benchmark Shows
To get a rough feel for the gap, my web-scraping-showcase repo has a compare-scroll script that extracts the same 100 quotes from quotes.toscrape.com (a public sandbox built for scraper testing) in two ways: Playwright scrolling the infinite-scroll page, and plain httpx calling the JSON endpoint behind it (/api/quotes?page=N, 10 pages).
| Extraction Strategy | Total Time (100 items, single run) |
Memory | Selector Breakage Risk |
|---|---|---|---|
| Playwright headless Chromium (scroll + CSS selectors) | ~4.80 s |
~180 MB (rough estimate, not measured) |
High (UI class renames) |
Direct JSON API via httpx |
~1.45 s |
< 15 MB (rough estimate, not measured) |
Low (structured JSON) |
That run came out about 3.3x faster for the JSON API. Treat it as an illustration, not a law: it is one small site, the API loop fetches pages one at a time, the Playwright time includes browser startup, and when I re-ran it on 2 Nov 2026 the ratio ranged from about 1.6x to 7.4x, mostly depending on network latency and Chromium cold start. The memory figures are rough estimates printed by the script, not measurements. The direction is consistent, though: no browser means less to download, less RAM and fewer moving parts, and a small container is usually enough for this kind of job.
The 4-Tier Extraction Waterfall
Before writing a single page.locator() or BeautifulSoup CSS query, inspect every target domain through four progressively heavier tiers:
Tier 1: application/ld+json (Schema.org) & Sitemaps
Any site that cares about Google Search rich snippets—Shopify stores, WooCommerce catalogs, healthcare directories, hospitality platforms, job boards—embeds machine-readable Schema.org entities right inside <script type="application/ld+json">.
Unlike visual DOM classes, JSON-LD tends to stay stable, because site owners have a reason to keep it working: Google reads it for rich results.
import json
from typing import Any, Iterator
from bs4 import BeautifulSoup
def iter_json_ld_entities(html: str) -> Iterator[dict[str, Any]]:
"""Extracts flattened Schema.org entities, handling nested @graph arrays."""
soup = BeautifulSoup(html, "html.parser")
for tag in soup.find_all("script", attrs={"type": "application/ld+json"}):
raw = (tag.string or tag.get_text() or "").strip()
if not raw:
continue
try:
payload = json.loads(raw)
except json.JSONDecodeError:
continue
stack = payload if isinstance(payload, list) else [payload]
while stack:
node = stack.pop()
if not isinstance(node, dict):
continue
if isinstance(node.get("@graph"), list):
stack.extend(node["@graph"])
if "@type" in node:
yield node
On many Shopify stores, combining the public /products.json?limit=250&page=1 endpoint with Product JSON-LD can give you SKUs, availability, prices and compare-at prices (plus currency and GTIN/MPN when the theme includes them) without executing a single line of client JavaScript. (My E-Commerce Price & Stock Monitor Actor currently implements only the JSON-LD half: it reads Product JSON-LD first and falls back to common HTML product cards. It does not call /products.json yet.)
Tier 2: SSR Hydration Payloads (__NEXT_DATA__, __NUXT__, RSC Flight)
Frameworks like Next.js, Nuxt, and Remix fetch data on the server to render SEO-friendly HTML, then serialize the exact server props into the document so client-side React doesn't have to refetch on mount.
-
Next.js Pages Router: Look for
<script id="__NEXT_DATA__" type="application/json">. Even better, once you grab thebuildIdfrom__NEXT_DATA__, subsequent page navigations are pure JSON endpoints at/_next/data/<buildId>/catalog.json?page=2! -
Next.js App Router (React Server Components): Inspect
self.__next_f.push([1, "..."])chunks or send theRSC: 1header to inspect Flight stream data.
Tier 3: Automated XHR / GraphQL API Discovery with Playwright
What if the initial HTML shell is empty and data loads dynamically on scroll or filter clicks? Instead of manually clicking through Chrome DevTools for 20 minutes, run a small Playwright XHR Sniffer once during development to map every JSON endpoint the page calls:
import asyncio
from playwright.async_api import async_playwright, Response
async def sniff_json_endpoints(target_url: str) -> None:
"""Captures background JSON XHR/Fetch endpoints, status codes, and payload keys."""
discovered: list[dict] = []
async def on_response(response: Response) -> None:
ctype = response.headers.get("content-type", "").lower()
if response.request.resource_type in ("xhr", "fetch") and "json" in ctype:
try:
body = await response.json()
top_keys = list(body.keys())[:6] if isinstance(body, dict) else f"list[{len(body)}]"
discovered.append({
"method": response.request.method,
"url": response.url,
"status": response.status,
"keys": top_keys,
"auth_headers": {
k: v for k, v in response.request.headers.items()
if k.lower() in ("authorization", "x-api-key", "x-csrf-token")
},
})
except Exception:
pass
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
page.on("response", on_response)
# Abort heavy visual assets so discovery finishes faster
await page.route(
"**/*.{png,jpg,jpeg,webp,svg,woff,woff2,mp4}",
lambda route: route.abort(),
)
await page.goto(target_url, wait_until="networkidle")
await browser.close()
for hit in discovered:
print(f"[{hit['status']}] {hit['method']} {hit['url']}\n Keys: {hit['keys']} | Auth: {hit['auth_headers']}")
if __name__ == "__main__":
asyncio.run(sniff_json_endpoints("https://example-directory.com/search"))
For example, when I built a personal-project Apify Actor for Malaysia's KPKT TEDUH housing-developer portal (apify-malaysia-kpkt-teduh-property-scraper), the page turned out to load its data from a clean backend JSON route, /api/developer-swasta?page=1, which returns 25 developers per page (as of November 2026) with their projects nested inside. That is much easier to work with than the rendered table.
Tier 4: Async httpx Client with Backoff & Hybrid Cookie Handoff
Once you know the JSON endpoint, don't hammer it with naive asyncio.gather(*[client.get(u) for u in thousand_urls]). Unbounded concurrency can trigger WAF rate limits (429 Too Many Requests), exhaust ephemeral TCP sockets, and get your IP banned.
A resilient async extractor needs three guardrails:
-
Bounded concurrency via
asyncio.Semaphore. -
Connection pooling via a single shared
httpx.AsyncClient(addhttp2=Trueif you installhttpx[http2]). -
Exponential backoff with full jitter respecting
Retry-Afterheaders.
import asyncio
import random
from typing import Any
import httpx
class ResilientJSONExtractor:
def __init__(
self,
concurrency: int = 8,
max_retries: int = 4,
cookies: dict[str, str] | None = None,
extra_headers: dict[str, str] | None = None,
) -> None:
self.sem = asyncio.Semaphore(concurrency)
self.max_retries = max_retries
self.headers = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
),
"Accept": "application/json, text/plain, */*",
**(extra_headers or {}),
}
self.cookies = cookies or {}
async def fetch_page(self, client: httpx.AsyncClient, url: str, params: dict[str, Any]) -> dict[str, Any]:
async with self.sem:
for attempt in range(1, self.max_retries + 1):
try:
resp = await client.get(url, params=params)
if resp.status_code == 200:
return resp.json()
if resp.status_code in (429, 502, 503, 504):
retry_after = float(resp.headers.get("Retry-After", 0))
backoff = max(retry_after, (2 ** (attempt - 1)) + random.uniform(0.1, 0.8))
await asyncio.sleep(backoff)
continue
resp.raise_for_status()
except (httpx.TransportError, httpx.TimeoutException):
if attempt == self.max_retries:
raise
await asyncio.sleep((2 ** (attempt - 1)) + random.uniform(0.1, 0.6))
raise RuntimeError(f"Exhausted {self.max_retries} retries for {url}")
async def paginate_cursor(self, base_url: str, max_pages: int = 50) -> list[dict[str, Any]]:
limits = httpx.Limits(max_keepalive_connections=20, max_connections=30)
records: list[dict[str, Any]] = []
async with httpx.AsyncClient(
headers=self.headers,
cookies=self.cookies,
timeout=httpx.Timeout(15.0, connect=5.0),
limits=limits,
follow_redirects=True,
) as client:
tasks = [self.fetch_page(client, base_url, {"page": p, "limit": 50}) for p in range(1, max_pages + 1)]
pages = await asyncio.gather(*tasks, return_exceptions=True)
for payload in pages:
if isinstance(payload, dict) and isinstance(payload.get("data"), list):
records.extend(payload["data"])
return records
One caveat: Retry-After can also be an HTTP date rather than a number of seconds, and float() will raise on that. The shared client in my web-scraping-showcase repo handles both forms.
When You Do Need Playwright: The Hybrid Session Handoff Pattern
Sometimes pure httpx hits a wall on request #1:
- Short-lived signed session cookies minted by a client-side JavaScript challenge.
- OAuth / SSO login flows with dynamic CSRF tokens or MFA steps.
-
GraphQL
persistedQueryhashes accompanied by rotatingx-client-transaction-idheaders.
Even then, you still don't need Playwright to render 10,000 detail pages! Use Hybrid Session Handoff:
- Launch Playwright once at job startup (a few seconds).
- Complete authentication or let the challenge cookie settle.
- Export
context.cookies()into adict({c["name"]: c["value"] for c in await context.cookies()}) and shut down Chromium immediately. - Feed those cookies into
ResilientJSONExtractorand run the remaining9,999requests as plain JSON calls, with no browser in memory.
If cookies expire every 20 minutes, wrap the handoff in an async token-refresh lock that only re-launches Playwright on a 401/403 status code.
Putting It Into Practice (Public Example Repos)
If you want to inspect or fork small example projects built around this idea, check out these repositories:
-
web-scraping-showcase: Small async Python examples: the Playwright-vs-JSON-API benchmark above, a Playwright XHR sniffer, and CSV/JSON/SQLite exporters. -
apify-ecommerce-price-and-stock-monitor: Async extractor that readsSchema.orgProductJSON-LD first and falls back to common HTML product cards (published on Apify Store; still new, with only a few runs so far).
Next time someone hands you a slow, crashing Selenium or Puppeteer script, pop open the Fetch/XHR tab first. Surprisingly often, the backend API you wish existed is already right there waiting for you.
I'm Trung Thanh Tran, a Python developer building small web scrapers, Apify Actors and automation tools.
- Want 6 ready-to-run Python 3.11 CLI engines (including a 4-Tier Universal Web & JSON Extractor) + 3 importable n8n/Make blueprints? Grab my Python, AI & n8n Scraper Toolkit Pro on Gumroad.
- Need a custom scraper, anti-bot audit, or end-to-end automation pipeline? Reach out on Upwork or Fiverr.
Top comments (2)
In paginate_cursor, gather(return_exceptions=True) lets a failed page disappear from the output while successful pages still look like a complete catalog. Could you return failed page IDs and a completeness flag alongside records, and keep the previous price-monitoring baseline whenever any page fails? Otherwise a transient timeout can look like products were removed.
Spot-on catch, Lucas — silent partial harvests are the #1 cause of false "removed/delisted SKU" alerts in diff-based price and stock monitors.
In the benchmark snippet I kept
paginate_cursorminimal to isolate transport overhead, warranted only for stateless one-off exports. In a stateful price-monitoring pipeline, you never wantreturn_exceptions=Trueto silently drop failed pages into the same bucket as a clean run.Here is the exact pattern I use in production to return explicit
failed_page_ids+is_complete, run a targeted retry pass on stragglers, and gate baseline pruning downstream:Then in the persistence layer, we split upserts from removals:
upsert): Updateprice/in_stock/last_seen_atfor every SKU present inresult.records, even on a partial run, so live price drops still trigger alerts immediately.result.is_complete(prune / mark_removed): Only diff against the previous baseline to mark unseen SKUs as delisted whenresult.is_completeisTrue(and total count doesn't trip a sanity-drop circuit breaker, e.g.,len(records) < 0.8 * prev_count). If any page failed, we retain the previous baseline for missing SKUs and queuefailed_pagesfor a narrow retry.Really appreciate you calling this out — I'll add a note right below that snippet so readers building stateful monitors don't trip on silent partial gathers!