When a scraped page comes back empty and render_js is the answer

#python #javascript #webscraping #spa

A scrape returns scrape_status: "success", statusCode: 200, and 300 characters of Markdown consisting of a nav bar and a footer. The page in your browser shows a full article. Nothing failed — the HTML the server sent genuinely contains no content, because the content is assembled by JavaScript after load.

This is the single most common false-success in web extraction, and the diagnosis takes one request.

Confirming the diagnosis

Fetch the raw HTML and look for the content, not for errors:

import os
import requests

API = "https://api.messora.dev"
HEADERS = {"X-API-Key": os.environ["MESSORA_API_KEY"]}


def diagnose(url: str, needle: str) -> None:
    resp = requests.post(
        f"{API}/scrape",
        headers=HEADERS,
        json={"url": url, "formats": ["raw", "markdown"]},
        timeout=90,
    )
    resp.raise_for_status()
    data = resp.json()

    html = data.get("rawHtml") or ""
    md = data.get("markdown") or ""

    print(f"status      : {data['scrape_status']} / {data['metadata']['statusCode']}")
    print(f"raw html    : {len(html)} chars")
    print(f"markdown    : {len(md)} chars")
    print(f"needle in raw: {needle.lower() in html.lower()}")


diagnose("https://example.com/app/article/42", needle="conclusion")

If the needle is absent from rawHtml, no extraction setting will find it. The text does not exist in the response — it arrives later, from an XHR call the static fetch never makes.

Enabling rendering

json={
    "url": url,
    "formats": ["markdown"],
    "render_js": True,
    "wait_for": 2500,
    "only_main_content": True,
    "timeout": 120,
}

render_js runs a real browser engine. wait_for adds a fixed pause in milliseconds after load, which is the blunt instrument that works when you cannot know which request populates the DOM.

Raise timeout alongside render_js. Rendering a heavy single-page application routinely takes 8 to 15 seconds, and the default 60-second budget covers that, but a client-side timeout of 30 will not.

Calibrating wait_for

Guessing wait_for wastes either time or content. Measure it once:

def calibrate(url: str, needle: str, candidates=(0, 1000, 2500, 5000, 8000)) -> int | None:
    for wait in candidates:
        resp = requests.post(
            f"{API}/scrape",
            headers=HEADERS,
            json={
                "url": url,
                "formats": ["markdown"],
                "render_js": True,
                "wait_for": wait,
                "timeout": 120,
            },
            timeout=180,
        )
        resp.raise_for_status()
        md = resp.json().get("markdown") or ""
        found = needle.lower() in md.lower()
        print(f"wait_for={wait:>5}ms  chars={len(md):>7}  found={found}")
        if found:
            return wait
    return None

Run it against three representative URLs from the same site and take the highest value that worked, plus a margin. A site with a consistent framework has a consistent hydration time; per-URL tuning is over-fitting.

Every calibration run costs 1 credit per attempt, so five candidates across three URLs is 15 credits to permanently settle a parameter you would otherwise guess at forever.

Do not enable it by default

Rendering costs wall-clock time on every page. On a /batch of 50 URLs where 45 are static, blanket render_js turns a two-minute job into something much longer for no additional content.

The cheaper pattern is static first, render on failure:

def scrape_adaptive(url: str, min_chars: int = 500) -> dict:
    """Static fetch first; re-fetch with rendering only when the result looks empty."""
    base = {"url": url, "formats": ["markdown"], "only_main_content": True}

    resp = requests.post(f"{API}/scrape", headers=HEADERS, json=base, timeout=90)
    resp.raise_for_status()
    data = resp.json()

    thin = len((data.get("markdown") or "").strip()) < min_chars
    if data["scrape_status"] == "success" and not thin:
        return data

    resp = requests.post(
        f"{API}/scrape",
        headers=HEADERS,
        json={**base, "render_js": True, "wait_for": 2500, "timeout": 120},
        timeout=180,
    )
    resp.raise_for_status()
    return resp.json()

This costs 2 credits on pages that need rendering and 1 on pages that do not. Against a corpus that is mostly static, that beats paying the rendering latency everywhere.

When rendering is not the problem

Two cases look identical from the outside and rendering does not fix either:

scrape_status: "blocked_antibot" — the fetch was refused. Rendering does not change that verdict; the request never got content to render.
Content behind authentication. A page that requires a session returns the login form, rendered perfectly. The Markdown will be a complete, correct rendering of a page you did not want.

Check scrape_status before reaching for render_js. It is only the right lever when the status is success and the body is thin — that specific combination is what "the HTML is empty on purpose" looks like from the client side.

DEV Community