DEV Community

Anakin
Anakin

Posted on

Browser automation is not always the right layer for data extraction

#ai #webscraping #api #automation

Your agent needs a product price, a listing count, or a search result. The first version opens Chromium, loads the page, waits for JavaScript, takes a DOM snapshot, and asks an LLM to find the value. It works in a demo. In production, the queue backs up, selectors drift, and half your retry budget goes to pages that rendered a cookie modal instead of the data.

The layer matters

Browser automation is useful, but it is an expensive default for data extraction.

A modern web page is usually not the source of truth. It is a client for backend APIs. The page loads HTML, runs JavaScript, calls internal endpoints, receives JSON, then renders a UI. If your task is “get the price” or “list the search results”, the browser is often doing work you do not need.

There are three common approaches:

  1. Run a browser and inspect the rendered page.
  2. Run a browser with saved site knowledge, such as selectors, workflows, and known endpoint hints.
  3. Call the underlying network API directly and parse structured data.

Browserbase Browse.sh fits the second category. It gives an agent reusable skills so it does not rediscover the same site on every run. That can reduce token use and make browser workflows more predictable, but Chromium and LLM inference still sit in the runtime path.

For covered sites, Wire exposes maintained network-layer actions so an agent can request structured data without rendering the page first.

What the browser path costs you

A typical browser extraction looks something like this:

import { chromium } from "playwright";

const browser = await chromium.launch();
const page = await browser.newPage();

await page.goto("https://example.com/search?q=mechanical+keyboard", {
  waitUntil: "networkidle",
  timeout: 30000
});

const items = await page.locator("[data-testid='result-card']").evaluateAll(cards =>
  cards.map(card => ({
    title: card.querySelector("h2")?.textContent?.trim(),
    price: card.querySelector(".price")?.textContent?.trim()
  }))
);

await browser.close();
console.log(items);
Enter fullscreen mode Exit fullscreen mode

This fails in familiar ways:

TimeoutError: page.goto: Timeout 30000ms exceeded
Enter fullscreen mode Exit fullscreen mode

or:

Error: strict mode violation: locator('.price') resolved to 2 elements
Enter fullscreen mode Exit fullscreen mode

or worse, it succeeds and returns the wrong thing because a sponsored result matched the same selector.

Adding site knowledge helps. A skill file can say “use this selector”, “avoid this path”, or “the first ID in this response is an offset, not a real listing ID.” That matters. It cuts down repeated exploration and makes the agent less likely to waste tokens reasoning about layout.

It does not remove browser startup time, page weight, bot challenges, modal handling, or the need to keep selectors current. If a site changes its rendered markup, your extraction can still break even if the underlying data contract stayed stable.

What network-layer extraction looks like

The network-layer version skips the rendered surface. Instead of asking “where is the price on the page?”, you ask “which request gave the page this data?”

In DevTools, that usually means:

  1. Open the Network tab.
  2. Filter for fetch or xhr.
  3. Perform the search or page action manually.
  4. Find the JSON response that contains the data.
  5. Reproduce that request with the required headers, cookies, query params, and body.

The runtime shape is closer to this:

curl -X POST https://api.openwire.sh/v1/wire/task \
  -H "X-API-Key: $ANAKIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "action_id": "amazon.search_products",
    "params": {
      "query": "mechanical keyboard",
      "limit": 10
    }
  }'
Enter fullscreen mode Exit fullscreen mode

A queued job response is easier to build around than a browser session:

{
  "job_id": "job_01j5x",
  "status": "queued",
  "poll": "/v1/wire/jobs/job_01j5x"
}
Enter fullscreen mode Exit fullscreen mode

Then the completed result is structured data:

{
  "status": "completed",
  "latency_ms": 178,
  "result": {
    "items": [
      {
        "title": "Keychron K2 Pro",
        "price": 99.99,
        "rating": 4.6,
        "reviews": 2340,
        "asin": "B0BJFDQ5KK"
      }
    ]
  }
}
Enter fullscreen mode Exit fullscreen mode

You can build this yourself for a specific site. The hard parts are request signing, session refresh, rate limits, schema drift, and detecting when an endpoint starts returning HTML, a login redirect, or a bot challenge instead of JSON.

That last failure mode is important. If your parser expects JSON and gets this, you want to fail loudly:

SyntaxError: Unexpected token '<', "<!doctype "... is not valid JSON
Enter fullscreen mode Exit fullscreen mode

Do not catch that and return an empty result set. Empty data and failed extraction are different states.

Auth changes the decision

Authentication often decides which layer you can use.

If the workflow requires a real user moving through a multi-step login, completing a form, or handling an interactive challenge, use a browser. Browser sessions preserve the same state the site expects from a human-operated client.

If the workflow is authenticated data retrieval, you may not need to expose raw credentials to the agent at all. For authenticated extraction, Wire uses named identities so code can reference an identity while credential resolution happens outside the agent prompt and response path.

If you build this internally, copy that separation. Store credentials in a vault, pass references through your job system, and make workers fetch short-lived session material at runtime. Do not put passwords, cookies, or bearer tokens in prompts. Also log auth failures distinctly:

401 Unauthorized: session expired
403 Forbidden: request signature invalid
200 OK text/html: login page returned instead of API response
Enter fullscreen mode Exit fullscreen mode

Those three cases need different retries.

A practical rule of thumb

Use network-layer extraction when the output is data, the site has a stable endpoint, and you can validate the response schema.

Use browser automation when the task is visual, interactive, outside your known integrations, or depends on rendered state that is not available from an API. Forms, checkout flows, CAPTCHA-adjacent paths, dashboards with heavy client state, and one-off unknown sites usually belong here.

The mistake is treating the browser as the universal interface. It is an interface, but it is not always the cheapest or most reliable one.

Before adding another browser worker, inspect the Network tab for the action you care about. If the data is already arriving as JSON, prototype the HTTP path, validate the schema, and measure p50 latency, p95 latency, failure rate, and cost per successful extraction against your browser version.

Top comments (0)