Scraping platform costs: measure successful rows, not browser minutes

A scraping job usually fails in boring ways: the browser hangs, a selector starts returning empty strings, a login expires, or the target site returns a captcha halfway through the run. The awkward part is that many platforms still bill you for the work done before the failure. If you run enough jobs, that difference shows up both in your invoice and in the amount of defensive code you need around the scraper.

Billing by compute time changes how you build

A lot of scraping platforms charge for runtime. Apify, for example, uses compute units: memory multiplied by time. A browser-heavy actor running for ten minutes with 2 GB of RAM consumes roughly a third of a compute unit before any actor-specific result fees.

That model is reasonable from the provider side. Chromium processes are expensive. Proxies cost money. Retries use resources.

But as the caller, you care about a different unit: did I get the rows I needed?

The hard part is that runtime billing makes cost hard to know before execution. A job that normally takes 30 seconds might take 8 minutes when a site slows down. A job that returns malformed data can still count as successful from the platform's point of view. A job that fails after rendering 200 pages still consumed browser time.

If your pipeline runs once a day, that may be fine. If it runs continuously, you probably want a local cost model that tracks outcomes, not just requests.

type ScrapeRun = {
  jobId: string;
  target: string;
  startedAt: string;
  finishedAt?: string;
  status: "queued" | "running" | "succeeded" | "failed";
  rowsExpected?: number;
  rowsReceived?: number;
  billedUnits?: number;
};

function isUsefulResult(run: ScrapeRun) {
  if (run.status !== "succeeded") return false;
  if (run.rowsExpected && (run.rowsReceived ?? 0) < run.rowsExpected * 0.9) {
    return false;
  }
  return (run.rowsReceived ?? 0) > 0;
}

function costPerUsefulRow(run: ScrapeRun) {
  if (!isUsefulResult(run)) return Infinity;
  return (run.billedUnits ?? 0) / (run.rowsReceived ?? 1);
}

This looks simple, but it catches a common lie in scraping metrics: "the run succeeded" does not mean "the data was usable."

For comparison, Wire prices web actions by successful outcome rather than browser runtime, which maps more closely to the metric most production pipelines already care about.

Treat empty data as a failure

The worst scraper failure is not a hard error. A hard error pages someone or triggers a retry. The worse version returns HTTP 200, exits cleanly, and gives you rows full of nulls.

That usually happens after a layout change:

<!-- Old markup -->
<span class="price">$129.00</span>

<!-- New markup -->
<div data-testid="product-price">$129.00</div>

Your selector still runs. It just finds nothing.

const price = await page.$eval(".price", el => el.textContent?.trim());

Depending on how the scraper is written, this might throw:

Error: failed to find element matching selector ".price"

Or it might quietly return undefined if the actor catches the exception and keeps going.

You should validate extracted data at the boundary, before it reaches your database or queue consumers.

import { z } from "zod";

const Product = z.object({
  title: z.string().min(1),
  price: z.string().regex(/^\$\d+(\.\d{2})?$/),
  url: z.string().url()
});

function validateProducts(rows: unknown[]) {
  const parsed = rows.map(row => Product.safeParse(row));
  const failures = parsed.filter(result => !result.success);

  if (failures.length > rows.length * 0.05) {
    throw new Error(`Too many invalid rows: ${failures.length}/${rows.length}`);
  }

  return parsed
    .filter(result => result.success)
    .map(result => result.data);
}

This gives you a clear failure mode. Instead of discovering bad data in a dashboard two weeks later, the pipeline rejects the run immediately.

Async jobs need idempotency and polling limits

Many scraping APIs use an async pattern: submit a job, get an ID, poll until the result is ready. That works well for long-running jobs, but you need to protect yourself from duplicate submissions and endless polling.

A minimal polling loop should have a timeout, backoff, and a stable idempotency key for the input.

import crypto from "node:crypto";

function idempotencyKey(input: unknown) {
  return crypto
    .createHash("sha256")
    .update(JSON.stringify(input))
    .digest("hex");
}

async function pollJob(jobId: string) {
  const deadline = Date.now() + 10 * 60 * 1000;
  let delayMs = 1000;

  while (Date.now() < deadline) {
    const res = await fetch(`https://api.example.com/jobs/${jobId}`);
    const job = await res.json();

    if (job.status === "succeeded") return job.result;
    if (job.status === "failed") throw new Error(job.error ?? "Job failed");

    await new Promise(resolve => setTimeout(resolve, delayMs));
    delayMs = Math.min(delayMs * 1.5, 15000);
  }

  throw new Error(`Timed out waiting for job ${jobId}`);
}

Without these guards, a network retry can submit the same expensive scrape twice, and a stuck job can keep a worker occupied forever.

Authenticated scraping has a different failure profile

Public pages mostly fail because markup changes or bot defenses get stricter. Logged-in scraping adds another set of problems: expired cookies, two-factor prompts, account locks, regional sessions, and captcha challenges.

A common workaround is to log in once, save cookies and local storage, then reuse them across runs. That works until the session expires or the target site binds it to a proxy, user agent, or device fingerprint.

The minimum you should store for authenticated scraping is:

{
  "cookies": [],
  "localStorage": {},
  "userAgent": "Mozilla/5.0 ...",
  "proxyRegion": "us-east",
  "createdAt": "2026-07-27T10:00:00Z",
  "expiresAt": "2026-07-28T10:00:00Z"
}

Do not treat this as a permanent credential. Treat it as a short-lived session artifact. Encrypt it, rotate it, and expect refresh failures.

Wire handles authenticated actions through named identities rather than asking each job to rebuild login state, which avoids a class of failures caused by repeatedly automating the login flow.

The practical comparison

When you evaluate a scraping platform, do not stop at "can it scrape this site?" Most tools can handle the happy path demo.

Ask these questions instead:

• Can I estimate the cost before the job starts?
• Do failed or partial runs still cost money?
• How do I detect valid-looking but wrong data?
• Who fixes the scraper when the site changes?
• Can I reuse authenticated sessions safely?
• What happens if I submit the same job twice?
• Does the platform fit batch jobs, continuous pipelines, or both?

If the answers are unclear, wrap the API with your own run ledger, schema validation, idempotency keys, and timeout handling before sending production traffic through it.