The "empty response" bug that quietly breaks a third of production scrapers (and a 20-line fix)

If you scrape at any scale, you've hit this: your job returns HTTP 200, your code saves a row, and the row is empty. No exception. No 403. Just silence — and a database full of nothing.

I lost a week to this before I understood it. Here's what's actually happening and a small detector that catches it before it corrupts your pipeline.

Why 200 does not mean "got the content"

Modern sites are increasingly client-side rendered. The server returns a tiny HTML shell — often literally <div id="root"></div> plus a script tag — and the real content is assembled in the browser by JavaScript after hydration.

Your requests.get() (or httpx, or aiohttp) fetches the shell. It gets a perfect 200. The DOM you expected was never there, because it doesn't exist until a browser runs the app.

r = httpx.get("https://example-shop.com/product/123")
print(r.status_code)        # 200
print(len(r.text))          # 812  ← a shell, not a product page

This is the failure mode that hurts the most, because it's invisible. A 403 you notice. An empty row you ship to production.

Three signals that the page never hydrated

You don't need a browser to detect this. You need to check three cheap things:

1. Visible-text ratio. Strip tags from the HTML and count characters. A real content page usually has a healthy text-to-HTML ratio; a shell has almost no text relative to its script payload.

import re

def visible_text_len(html: str) -> int:
    # remove script/style, then strip tags
    html = re.sub(r"<(script|style)[^>]*>.*?</\1>", "", html, flags=re.S | re.I)
    text = re.sub(r"<[^>]+>", " ", html)
    return len(re.sub(r"\s+", " ", text).strip())

1. Known app-shell markers. A surprisingly large share of SPA failures are detectable by the presence of an empty mount node and nothing else:

SHELL_MARKERS = [
    r'<div id="root"[^>]*>\s*</div>',
    r'<div id="app"[^>]*>\s*</div>',
    r'<div id="__next"[^>]*>\s*</div>',
    r'window\.__NUXT__',
    r'__NEXT_DATA__',
]

1. Expected-field absence. The strongest signal: you know what a valid page looks like. If your CSS/selector for the price or title returns nothing, the fetch failed regardless of status code.

The detector

Combine the three into one gate that decides how to fetch:

import re
import httpx

SHELL_MARKERS = re.compile(
    r'<div id="(root|app|__next)"[^>]*>\s*</div>|window\.__NUXT__|__NEXT_DATA__',
    re.I,
)


def looks_unhydrated(html: str, min_text: int = 400) -> bool:
    """True if the HTML looks like an un-rendered SPA shell."""
    if not html:
        return True
    if SHELL_MARKERS.search(html):
        return True
    if visible_text_len(html) < min_text:
        return True
    return False


def fetch(url: str) -> str:
    # fast path: plain HTTP
    r = httpx.get(url, follow_redirects=True, timeout=15,
                  headers={"User-Agent": "Mozilla/5.0"})
    if r.status_code == 200 and not looks_unhydrated(r.text):
        return r.text

    # slow path: the page needs a real browser to hydrate
    return fetch_with_browser(url)   # Playwright / your headless engine

That's the whole idea: use the cheap engine until it tells you it can't work, then escalate. In practice this keeps ~70% of requests on the fast path and only pays the headless-browser cost where it's actually needed.

The gotchas nobody mentions

• min_text=400 is a guess — calibrate it. Run the detector across a sample of your own known-good and known-bad URLs and pick the threshold that separates them. A news article and a product card have very different baselines.
• __NEXT_DATA__ is not always a failure. Some Next.js sites server-render and embed the data blob. Check the visible-text ratio too before escalating, or you'll double-fetch everything.
• Don't escalate on a 403/429. Those are bot-management blocks, not hydration failures — a browser fetch may not help and you'll just get blocked slower. Handle them separately (backoff, proxy, or skip).
• Escalation is not free. A headless browser is 50–200× the cost of an HTTP request. Gate it tightly.
• Log the decision. Emit engine=http vs engine=browser per URL. The day your fast path silently starts failing is the day you want that metric.

Why this matters beyond scraping

Any system that fetches a page and then reasons about it — AI agents that read websites, RAG pipelines that ingest docs, monitoring bots — inherits this exact bug. An LLM given an empty shell will happily hallucinate a product page from the URL alone, and now your "automation" is confidently wrong.

Checking for the empty response before you use it is the cheapest reliability upgrade you can ship this week.

---

I packaged the full version of this — the dual-engine fetcher (async HTTPX fast path + stealth Playwright escalation), the calibrated detector, dedupe, and a styled Excel export with clickable contact links — as a self-hosted toolkit here (pay what you want):

https://ancuboy.gumroad.com/l/python-scraper-toolkit

It's the tool I built to stop paying ~$99/month for a cloud scraper that broke on exactly this problem. Feedback welcome — the engine-routing heuristics are the part I'm still tuning.

Comments

[Saad | Data APIs]

Good detector for SPA shells. A cousin of the same silent failure shows up on JSON job-board endpoints: Greenhouse's boards-api returns 200 with full job objects but no content field unless you pass ?content=true. Status looks perfect, your pipeline writes rows, and every description is blank until you notice days later.

I gate that the same way you gate shells — require an expected field (content non-empty, or at least absolute_url) before marking the fetch healthy. Visible-text ratio ≈ "required field present".