How to Read JavaScript-Rendered Pages from an AI Agent (When Extract Comes Back Empty)

If you want to read a single-page app or JavaScript page from an AI agent, the first surprise is usually an empty result. Your agent calls an extractor, gets back a page title and one line telling it to enable JavaScript, and then confidently reports that "the page has no content." Nothing failed. The extractor did its job on HTML that simply had no text in it yet.

This post is about that moment: how to tell, in code, that an Extract result is a JavaScript shell rather than a real page, and what your agent should do next. We cover the detection checks, a small Python function that tries Web Extract first and falls back to Web Render only when needed, the error codes each tool returns, and when Render is the wrong answer. Each call is $0.005 in USDC on the Pocket Agentic Portal, paid over x402 on Base or MPP, with no signup.

If you want the TypeScript version of the full scraping pipeline, see How to scrape JavaScript-heavy pages for AI agents. This one focuses on the read decision inside an agent's tool.

Why Extract comes back empty on an SPA

Extract fetches the HTML a server sends and turns it into clean markdown. On a server-rendered page, that HTML already holds the article, so the markdown is complete.

A single-page app (React, Vue, Svelte, Angular and the like) often sends a near-empty document: a <head>, a <div id="root"></div>, a <noscript> notice, and a bundle of JavaScript that builds the page in the browser. Extract does not run that JavaScript, so it converts what it was given. The result is thin or empty, and it is accurate for the HTML it received.

Render loads the URL in headless Chromium, lets the scripts run, and returns what the browser ended up with: markdown, text, html, title, links and an optional screenshot. That is the tool for client-rendered pages. It also costs the same $0.005 per call, which is what makes "try Extract, then Render" a sensible default instead of rendering everything.

How to detect a thin or empty Extract result

Extract will not tell you "this page needed JavaScript." Your agent has to infer it. Three signals cover most real cases.

1. A length threshold. The simplest check: if the markdown is shorter than a few hundred characters, treat it as suspicious. Pick the number yourself and tune it for your sources; it is a local heuristic, not a service field. Some pages really are short (a contact page, a redirect notice), so a short result is a trigger for one Render call, not proof.

2. No-JavaScript markers. SPAs often ship a <noscript> message that survives conversion to markdown. Look for phrases such as "enable JavaScript", "requires JavaScript", "JavaScript is disabled", or "You need to enable JavaScript to run this app" (the default text in many React starter templates). If the page's own text says it needs a browser, believe it.

3. An empty root container. The classic SPA shell is a single <div id="root"></div> or <div id="app"></div> with nothing inside. That div produces no text at all in markdown, so in practice it looks like a title with no body, or a body that is only navigation labels and a footer. Treat "title present, body essentially empty" as the markdown signature of an empty root div.

Two things to avoid: do not retry Extract on a thin result (it fetches the same HTML again and you pay again), and do not treat a website error as "thin." Errors are a separate branch, covered below.

A Python fallback: Extract first, then Render

The function below uses the same call_service wrapper as the deep research agent guide: pay_fetch is your x402 v2 client (stubbed there), and call_service returns either a portal failure or the delivered data. Request and response shapes match that post: both tools take url, formats and max_chars, and return url (final URL after redirects), title, markdown and error (null on success).

# read_js_page.py
from __future__ import annotations

from typing import Any

# call_service, PortalFailure and Delivered come from research_agent.py
# in the deep research agent guide (Step 1). Requests go to
# https://agent.pocket.network/v1/<service-id>/v1/<tool>.
from research_agent import call_service

THIN = 400  # local heuristic, tune for your sources
JS_HINTS = (
    "enable javascript",
    "requires javascript",
    "javascript is disabled",
    "you need to enable javascript",
)

REQUEST = {"formats": ["markdown"], "max_chars": 8000}


def looks_like_js_shell(page: dict[str, Any]) -> bool:
    if page.get("error"):
        return False  # an error is not a thin page; handle it separately
    md = (page.get("markdown") or "").strip()
    lower = md.lower()
    if len(md) < THIN:
        return True  # covers the empty <div id="root"></div> case: no body text
    return any(h in lower for h in JS_HINTS)


def skip(url: str, code: str) -> dict[str, Any]:
    return {"status": "skip", "url": url, "code": code}


def from_portal_failure(url: str, failure: Any) -> dict[str, Any]:
    # robots.txt refusal: 400 UPSTREAM_REJECTED via the portal, 422
    # ROBOTS_DISALLOWED when calling Render directly. Not charged.
    if failure.code in ("UPSTREAM_REJECTED", "ROBOTS_DISALLOWED"):
        return skip(url, failure.code)
    status = "retry_later" if failure.retryable else "skip"
    return {"status": status, "url": url, "code": failure.code}


def read_js_page(url: str) -> dict[str, Any]:
    """Extract first; Render once if the result looks like a JS shell.

    Cost: 1 call ($0.005) for static pages, 2 calls ($0.01) for SPAs.
    """
    first = call_service("agentsearch-web-extract-v1", "/v1/extract", {"url": url, **REQUEST})
    if first.kind != "data":
        return from_portal_failure(url, first)

    page = first.data
    err = page.get("error")
    if err:
        # TARGET_TIMEOUT: Extract had no complete answer in time.
        # UNSUPPORTED_CONTENT: not an HTML page (a PDF, an image, ...).
        # The call was delivered and counts. Do not loop on the same URL.
        return skip(page.get("url") or url, str(err.get("code")))

    if not looks_like_js_shell(page):
        return {"status": "ok", "url": page.get("url") or url,
                "title": page.get("title"), "markdown": page["markdown"].strip()}

    second = call_service("agentsearch-web-render-v1", "/v1/render", {"url": url, **REQUEST})
    if second.kind != "data":
        return from_portal_failure(url, second)

    page = second.data
    err = page.get("error")
    md = (page.get("markdown") or "").strip()
    final_url = page.get("url") or url

    if err:
        if err.get("code") == "TARGET_DEADLINE" and md:
            # Render-only timeout: keep the text that rendered in time.
            return {"status": "ok", "url": final_url, "title": page.get("title"),
                    "markdown": md, "note": "partial: render hit its deadline"}
        status = "retry_later" if err.get("retryable") else "skip"
        return {"status": status, "url": final_url, "code": str(err.get("code"))}

    if len(md) < THIN:
        return skip(final_url, "EMPTY_PAGE")  # nothing public to read; do not pay again
    return {"status": "ok", "url": final_url, "title": page.get("title"), "markdown": md}

Hand the returned dict straight back to the model as the tool result. skip tells it to choose another source; retry_later means "on another turn," never "loop now." EMPTY_PAGE is a local code, not one the service returns.

What each error code means here

• TARGET_TIMEOUT (Extract). The site did not give Extract a complete answer within its deadline. Skip the page or pick another result.
• UNSUPPORTED_CONTENT (Extract). The URL is not HTML Extract can convert. Rendering it will not help.
• TARGET_DEADLINE (Render only). Render hit its hard deadline. If some content rendered in time, you get HTTP 200 with partial markdown: keep it and label it partial. If nothing rendered, you get HTTP 422 TARGET_DEADLINE, which is retryable and not charged.
• ROBOTS_DISALLOWED / UPSTREAM_REJECTED. The site's robots.txt blocks Render. More below.

Branch on code and retryable, never on the message text. The full list lives in Error handling for AI agent web tools.

The cost note

A static page costs one Extract call: $0.005. A JavaScript page costs one Extract plus one Render: 2 calls, $0.01. A robots.txt refusal costs nothing. If you already know a host is client-rendered (it came back thin last time), call Render directly and save the first call. Put a hard cap on delivered calls per task in code so the bill does not depend on the model's judgment.

Respecting robots.txt (and not paying for refusals)

Render honors robots.txt. If a site disallows its user agent, Render refuses the URL before fetching anything. Called directly, that is HTTP 422 with ROBOTS_DISALLOWED:

{"error": {"code": "ROBOTS_DISALLOWED", "message": "robots.txt disallows this URL for AgentSearchRender.", "retryable": false, "request_id": "…"}}

Through the Pocket portal it surfaces as HTTP 400 UPSTREAM_REJECTED, and there is no charge. The right agent behavior is to accept the answer: the site has asked bots not to load that page, so move on to another source. Do not retry, rotate tools, or try to get around the block.

When not to use Render

Render is a fallback, not a default. Skip it when:

• Extract already returned real text. If the markdown reads like the article, you are done at one call.
• The page needs a login. Render does not sign in. A login wall renders as a login form.
• The content is not HTML. A PDF or image that returned UNSUPPORTED_CONTENT from Extract will not improve in a browser.
• The site blocks bots. A robots.txt refusal or a bot check is the site's answer. Choose a different source.
• There is a public JSON or feed URL. Many SPAs load their text from a public API endpoint. If you already know it, a plain fetch of that is simpler.
• You do not have a URL yet. Use Web Search first, then read only the hits whose snippets look relevant.

Using it without writing a client

If your agent runs in an MCP client, @agentsearchhq/agentsearch-mcp 1.3.x exposes Search, Extract and Render as tools, with Search free out of the box. Framework users can start from langchain-agentsearch (PyPI), @agentsearchhq/langchain (npm), crewai-agentsearch, llama-index-tools-agentsearch, openai-agents-agentsearch or n8n-nodes-agentsearch. Whatever the wrapper, give the model the same rule in its instructions: Extract first, Render only when the markdown is thin or asks for JavaScript.

FAQ

How do I read a single-page app from an AI agent?

Call Extract first. If the markdown is very short, contains an "enable JavaScript" notice, or is a title with no body, call Render once on the same URL. Render runs the page in headless Chromium and returns markdown.

Why does Extract return empty content on some pages?

Client-rendered pages ship an empty container plus JavaScript. Extract converts the HTML it receives and does not run scripts, so the result is thin.

How much does the fallback cost?

$0.005 for a page Extract can read, $0.01 (2 calls) when Render is needed, and nothing when robots.txt refuses Render.

What if Render times out?

If some content rendered, you get HTTP 200 with TARGET_DEADLINE and partial markdown; keep it and mark it incomplete. If nothing rendered, you get HTTP 422 TARGET_DEADLINE, which is retryable and not charged.

Can Render read pages behind a login or a bot check?

No. It does not sign in, and it respects robots.txt. Pick another source.

Get started

Drop read_js_page into any agent tool that already calls Extract. Start with Web Extract, fall back to Web Render, and find URLs with Web Search. More guides are on the AgentSearch blog.

---

Originally published on the AgentSearch blog.