We Scaled Headless Browsers. That Was the Wrong Optimization.

Our first JavaScript scraping architecture was easy to explain:

URL → headless browser → wait → extract

It also wasted memory, increased latency, and made every failure look like “the browser timed out.”

The surprising fix was not more browser workers.

The scalable unit is the rendering decision, not the browser.

Most URLs do not deserve the most expensive retrieval path. The production system should decide when rendering is necessary, choose a completion signal, validate the result, and preserve enough evidence to explain a rejection.

How we knew static HTML was not enough

A page can return 200 OK while the original HTML contains only an application shell:

<div id="app"></div>
<script src="/assets/app.js"></script>

The price, article, or table appears only after JavaScript runs. But the presence of a script tag does not prove that a browser is required—almost every modern page has scripts.

We started checking the data we actually needed:

def needs_rendering(html: str, required_markers: list[str]) -> bool:
    text = html.lower()
    return not all(marker.lower() in text for marker in required_markers)

The first request is cheap. If the required content is already present, we parse it. If not, the URL becomes a browser candidate.

The architecture that held up

admission + dedupe
        ↓
static fetch
   ├─ valid → extract → validate → store
   └─ incomplete → browser queue
                       ↓
                 render + wait
                       ↓
                 validate → store

Each stage has one job.

Admission prevents duplicate cost

Normalize URLs, apply crawl boundaries, reject disallowed schemes, and generate an idempotency key before scheduling work. Otherwise, tracking parameters and retries can create several browser jobs for the same page.

The static path is not a fallback

Static retrieval should be a first-class path with its own success metrics. It is usually faster and easier to debug than a browser.

The browser queue protects memory

Browser concurrency should be bounded by measured memory use, not by an arbitrary worker count. Ten pages with large client bundles can consume more memory than one hundred small pages.

Validation happens after rendering

networkidle does not mean “the product price is correct.” The page may keep analytics connections open forever, or become quiet before a delayed component appears.

Use a completion signal tied to the task:

• a required selector exists;
• a data attribute has a valid value;
• a loading indicator disappears;
• a specific response completes;
• or a semantic content check passes.

The worker loop

The useful part of a browser worker is not goto(). It is the terminal-state discipline around it.

async def process(job, page):
    try:
        await page.goto(job.url, wait_until="domcontentloaded", timeout=20_000)
        await page.wait_for_selector(job.required_selector, timeout=10_000)

        html = await page.content()
        record = extract(html)
        validate(record)

        return {"state": "accepted", "record": record}
    except TimeoutError:
        return {"state": "render_timeout", "retryable": True}
    except ValidationError as exc:
        return {"state": "rejected_content", "reason": str(exc)}

A timeout and a semantically wrong page are different failures. Retrying both identically hides the real problem.

Where managed crawling fits

Operating Chromium, queues, retries, proxy routing, artifact storage, and task polling can become its own platform. Nstdata Crawl is useful when a team wants those collection primitives behind a managed interface, including static and browser-backed retrieval plus multiple output formats.

The managed API still cannot define correctness for your domain. A rendered page is an artifact; your application decides whether it is acceptable evidence.

The metrics that explained our bill

We stopped looking only at request count and added:

static acceptance rate
browser escalation rate
browser startup latency
render timeout rate
semantic rejection rate
memory per active context
cost per accepted page

The browser escalation rate was the most actionable. If it rises after a parser release, the problem may be our static detector—not the websites.

Final takeaway

Scaling headless browsing is partly an infrastructure problem. Deciding which pages deserve a browser is a data-quality problem.

Fetch cheaply first. Escalate deliberately. Wait for evidence, not generic quiet. And never confuse a rendered page with a validated record.

What percentage of your current browser jobs could have been accepted from the original HTML?