Browser automation vs network calls for sites without public APIs

You need data from a portal that never shipped a public API. The first version is usually a browser script: open the page, log in, click the button, scrape the table. It works until the site ships a harmless UI change and your job starts failing with TimeoutError: locator.click: Timeout 30000ms exceeded because the button text changed.

The real choice is the layer

There are two common ways to automate a site that only exposes a web UI.

Browser automation operates at the presentation layer. Tools like Playwright, Puppeteer, Selenium, and managed platforms such as CloudCruise drive a real browser. They click buttons, type into inputs, wait for navigation, handle downloads, and read data from the DOM.

Network automation operates one layer lower. Instead of clicking the UI, you inspect the requests the frontend already makes and call those endpoints directly. The page might not have a public API, but the JavaScript app usually talks to some backend API.

The difference matters because the failure modes are different.

A browser script depends on the page structure. If the app changes a label, moves a button, delays hydration, or swaps a table for virtualized rows, your script can break even if the underlying business action still works.

A network call depends on request shape, auth, and backend behavior. If the endpoint path changes, a CSRF token expires, the API starts requiring a new header, or the response schema changes, your integration breaks there instead.

For example, Wire is a maintained network-layer catalog for calling site actions directly when the useful behavior sits behind the page rather than in a public API.

What browser automation failure looks like

This is a typical Playwright flow:

import { chromium } from 'playwright';

const browser = await chromium.launch();
const page = await browser.newPage();

await page.goto('https://example-portal.test/login');
await page.getByLabel('Email').fill(process.env.PORTAL_USER);
await page.getByLabel('Password').fill(process.env.PORTAL_PASS);
await page.getByRole('button', { name: 'Sign in' }).click();

await page.getByRole('link', { name: 'Reports' }).click();
await page.getByRole('button', { name: 'Download CSV' }).click();

await browser.close();

This can fail even when the portal still works for a person:

TimeoutError: locator.click: Timeout 30000ms exceeded.
Call log:
  - waiting for getByRole('button', { name: 'Download CSV' })

Maybe the product team renamed the button to Export. Maybe the button is now an icon with no accessible name. Maybe a cookie banner covers it. The backend endpoint that generates the CSV may not have changed at all.

The browser approach is still the right tool when you need the actual user flow. Login with SSO, TOTP prompts, healthcare payer portals, EHR systems, visual confirmations, file pickers, and workflows that only exist as UI interactions all fit here. CloudCruise, for example, focuses on managed browser workflows for healthcare portals, including credential handling and 2FA support. That is a real use case, not something you should replace with a fragile copied request.

What network automation failure looks like

If you inspect DevTools while clicking Download CSV, you might find a request like this:

curl 'https://example-portal.test/api/reports/claims?month=2026-01' \
  -H 'authorization: Bearer eyJhbGciOi...' \
  -H 'x-csrf-token: 7d1b0...' \
  -H 'accept: application/json'

A safer implementation checks status codes and schema instead of assuming the request worked:

async function fetchClaimsReport({ month, token, csrf }) {
  const res = await fetch(
    `https://example-portal.test/api/reports/claims?month=${encodeURIComponent(month)}`,
    {
      headers: {
        authorization: `Bearer ${token}`,
        'x-csrf-token': csrf,
        accept: 'application/json'
      }
    }
  );

  if (res.status === 401 || res.status === 403) {
    throw new Error(`auth failed: ${res.status}`);
  }

  if (res.status === 429) {
    throw new Error('rate limited');
  }

  if (!res.ok) {
    throw new Error(`report request failed: ${res.status}`);
  }

  const data = await res.json();

  if (!Array.isArray(data.claims)) {
    throw new Error('unexpected response shape: claims is missing');
  }

  return data.claims;
}

This version avoids DOM breakage, browser startup time, screenshots, and selector maintenance. It also gives you clearer operational errors: 401, 403, 429, or unexpected response shape instead of a generic click timeout.

But it has edges. You must understand authentication. You need to respect rate limits. You may need to refresh CSRF tokens. Some endpoints are intentionally tied to short-lived browser sessions. Some sites prohibit automated access in their terms. Direct calls are not automatically safer or more acceptable just because they are faster.

Long-running actions need a job pattern

Browser jobs and network jobs can both outlive a single HTTP request. A report might take 45 seconds to generate. A search might fan out across multiple pages. Do not keep one request open forever if the caller can lose the connection.

Use a task creation endpoint and poll by job ID:

async function waitForJob(jobId, getStatus, { timeoutMs = 120000 } = {}) {
  const started = Date.now();
  let delay = 1000;

  while (Date.now() - started < timeoutMs) {
    const job = await getStatus(jobId);

    if (job.status === 'succeeded') return job.result;
    if (job.status === 'failed') throw new Error(job.error || 'job failed');

    await new Promise(resolve => setTimeout(resolve, delay));
    delay = Math.min(delay * 1.5, 5000);
  }

  throw new Error(`job timed out: ${jobId}`);
}

Wire models long-running site actions this way: create a task, keep the returned job ID, then poll until the action returns structured JSON or a failure.

How to choose

Use browser automation when:

• The workflow requires real user interaction, MFA, SSO, or visual confirmation
• The site does not expose a clean backend request for the action
• You need screenshots, video replay, or DOM-level debugging
• Compliance requirements point you toward a managed browser platform built for that domain

Use network automation when:

• The frontend calls a stable JSON endpoint for the data or action you need
• You care about latency and concurrency more than visual fidelity
• You want structured errors based on HTTP status and response schema
• UI changes should not break the integration

For your next portal automation, record one successful manual run with DevTools open. If the important action maps to a clear backend request, prototype the network call and validate auth, rate limits, and response shape. If it does not, write the browser workflow and make selector failure an expected operational case rather than a surprise.

Comments

[Double CHEN]

The job-ID example separates request lifetime from work lifetime. One more distinction helps: a client timeout does not prove the report job stopped. Persist the job ID before polling, and resume status checks after reconnecting instead of creating another report. Otherwise a retry can duplicate an expensive action.