DEV Community: Anakin

Session Reuse and Browser Mode Decisions for Web Data Pipelines

Anakin — Wed, 29 Jul 2026 10:00:01 +0000

A scraper that works on public HTML often fails the first time you point it at real customer data. Instead of the dashboard content, you index a login page. Instead of a product catalog, you get region-specific defaults. Instead of JSON, your parser throws because the page is just a JavaScript shell.

These are not parsing problems first. They are state and execution problems.

Know what kind of page you are fetching

Before reaching for a browser, classify the failure.

Common symptoms:

Symptom	What it usually means
`401 Unauthorized`	Missing or expired credentials
HTML contains `Sign in` or `Log in`	You fetched the login state, not the authenticated state
`403 Forbidden`	Blocked client, wrong region, or missing auth
Empty `<div id="root"></div>` page	Content renders client-side after hydration
Different price than expected	Geo-routing or account tier changed the response
`Unexpected token < in JSON at position 0`	Code expected JSON but received HTML, often an error page

Do this classification before adding retries. Retrying a login page gives you the same login page.

Reuse sessions only inside clear boundaries

Authenticated scraping needs session state: cookies, local storage, sometimes an assigned exit region. Re-authenticating for every fetch wastes time and can trigger security systems. Reusing sessions too broadly creates a worse problem: data leaks across users or tenants.

A reasonable policy is:

Same domain, same user, same auth context: reuse the session.
Same domain, different role or account: create a new session.
Different customer or tenant: always create a new session.
Logged-in and logged-out views: treat them as separate sessions.
Expired or revoked credentials: delete the session and require re-authentication.

You can encode that policy directly instead of leaving it to caller discipline:

type SessionKeyInput = {
  tenantId: string;
  userId: string;
  domain: string;
  authContext: 'anonymous' | 'customer' | 'admin';
  country?: string;
};

function sessionKey(input: SessionKeyInput) {
  return [
    input.tenantId,
    input.userId,
    input.domain,
    input.authContext,
    input.country ?? 'default'
  ].join(':');
}

function shouldReuseSession(a: SessionKeyInput, b: SessionKeyInput) {
  return sessionKey(a) === sessionKey(b);
}

The key point is tenant isolation. If two customers use the same upstream SaaS app, they still need separate sessions. Shared cookies are a security bug, not an optimization.

Set an expiration policy too. Ninety days may be fine for some internal tools. Banking or healthcare workflows may need much shorter retention. Whatever you choose, make deletion explicit and irreversible from the application point of view.

Pick standard HTTP before browser automation

Browser mode costs more because it does more. It starts a browser context, runs JavaScript, waits for the page to settle, and sometimes interacts with the page. That is useful when the content only exists after client-side execution. It is wasteful when the server already returns usable HTML.

A good default decision tree looks like this:

type FetchMode = 'http' | 'browser';

type PageSignals = {
  status: number;
  html: string;
  needsScroll?: boolean;
  hasCaptcha?: boolean;
};

function chooseFetchMode(signals: PageSignals): FetchMode {
  if (signals.status === 403) return 'browser';
  if (signals.hasCaptcha) return 'browser';
  if (signals.needsScroll) return 'browser';

  const looksLikeJsShell =
    signals.html.includes('<div id="root"></div>') ||
    signals.html.includes('<div id="app"></div>');

  if (looksLikeJsShell && signals.html.length < 5000) {
    return 'browser';
  }

  return 'http';
}

This is intentionally conservative. A React app can still server-render useful HTML, and a short HTML document is not always a failure. Log the decision and sample responses so you can tune the heuristics per domain.

If your extraction provider can authenticate and extract at the network layer, Wire belongs on the non-browser side of this decision: use browser automation only when page behavior requires a real browser.

Keep geo-routing part of the session identity

Region-specific content creates subtle bugs. The request succeeds, the parser works, and the model still answers incorrectly because it saw the US catalog instead of the India catalog.

Treat country or region as part of the request and session identity:

{
  "url": "https://example.com/pricing",
  "session_id": "tenant_42:user_7:example.com:customer:in",
  "country": "in"
}

Do not reuse a country=us authenticated session for a country=in fetch unless you have verified the upstream application ignores region. Many sites bind catalog, currency, legal text, and inventory to both account and IP region.

Cache by everything that changes the response

Caching only by URL is usually wrong for authenticated or regional pages.

Use a cache key that includes the meaningful request dimensions:

import crypto from 'node:crypto';

function cacheKey(input: {
  url: string;
  tenantId: string;
  authContext: string;
  country?: string;
  headers?: Record<string, string>;
}) {
  const material = JSON.stringify({
    url: input.url,
    tenantId: input.tenantId,
    authContext: input.authContext,
    country: input.country ?? null,
    acceptLanguage: input.headers?.['accept-language'] ?? null
  });

  return crypto.createHash('sha256').update(material).digest('hex');
}

For public pages, a URL-only cache may be enough. For logged-in dashboards, pricing pages, or catalogs, include tenant, auth context, country, and any header that changes the response.

Monitor the failures that indicate wrong mode

Track more than success rate. A request that returns 200 OK with a login page is still a failed extraction.

Useful production checks include:

Percentage of fetched pages that contain login markers.
Percentage of pages with very low text length.
Browser fallback rate per domain.
Session expiration rate.
Region mismatch reports from downstream validation.

If browser fallback climbs from 5% to 60% for a domain, something changed. Maybe the site moved rendering client-side. Maybe it started blocking your HTTP client. Either way, you want to notice before stale or empty documents enter the index.

The practical next step is to add response classification before parsing: detect login pages, JS shells, region mismatches, and empty content, then choose session reuse or browser execution based on those signals.

The full breakdown is here if you want the complete picture.

Scraping platform costs: measure successful rows, not browser minutes

Anakin — Tue, 28 Jul 2026 10:00:02 +0000

A scraping job usually fails in boring ways: the browser hangs, a selector starts returning empty strings, a login expires, or the target site returns a captcha halfway through the run. The awkward part is that many platforms still bill you for the work done before the failure. If you run enough jobs, that difference shows up both in your invoice and in the amount of defensive code you need around the scraper.

Billing by compute time changes how you build

A lot of scraping platforms charge for runtime. Apify, for example, uses compute units: memory multiplied by time. A browser-heavy actor running for ten minutes with 2 GB of RAM consumes roughly a third of a compute unit before any actor-specific result fees.

That model is reasonable from the provider side. Chromium processes are expensive. Proxies cost money. Retries use resources.

But as the caller, you care about a different unit: did I get the rows I needed?

The hard part is that runtime billing makes cost hard to know before execution. A job that normally takes 30 seconds might take 8 minutes when a site slows down. A job that returns malformed data can still count as successful from the platform's point of view. A job that fails after rendering 200 pages still consumed browser time.

If your pipeline runs once a day, that may be fine. If it runs continuously, you probably want a local cost model that tracks outcomes, not just requests.

type ScrapeRun = {
  jobId: string;
  target: string;
  startedAt: string;
  finishedAt?: string;
  status: "queued" | "running" | "succeeded" | "failed";
  rowsExpected?: number;
  rowsReceived?: number;
  billedUnits?: number;
};

function isUsefulResult(run: ScrapeRun) {
  if (run.status !== "succeeded") return false;
  if (run.rowsExpected && (run.rowsReceived ?? 0) < run.rowsExpected * 0.9) {
    return false;
  }
  return (run.rowsReceived ?? 0) > 0;
}

function costPerUsefulRow(run: ScrapeRun) {
  if (!isUsefulResult(run)) return Infinity;
  return (run.billedUnits ?? 0) / (run.rowsReceived ?? 1);
}

This looks simple, but it catches a common lie in scraping metrics: "the run succeeded" does not mean "the data was usable."

For comparison, Wire prices web actions by successful outcome rather than browser runtime, which maps more closely to the metric most production pipelines already care about.

Treat empty data as a failure

The worst scraper failure is not a hard error. A hard error pages someone or triggers a retry. The worse version returns HTTP 200, exits cleanly, and gives you rows full of nulls.

That usually happens after a layout change:

<!-- Old markup -->
<span class="price">$129.00</span>

<!-- New markup -->
<div data-testid="product-price">$129.00</div>

Your selector still runs. It just finds nothing.

const price = await page.$eval(".price", el => el.textContent?.trim());

Depending on how the scraper is written, this might throw:

Error: failed to find element matching selector ".price"

Or it might quietly return undefined if the actor catches the exception and keeps going.

You should validate extracted data at the boundary, before it reaches your database or queue consumers.

import { z } from "zod";

const Product = z.object({
  title: z.string().min(1),
  price: z.string().regex(/^\$\d+(\.\d{2})?$/),
  url: z.string().url()
});

function validateProducts(rows: unknown[]) {
  const parsed = rows.map(row => Product.safeParse(row));
  const failures = parsed.filter(result => !result.success);

  if (failures.length > rows.length * 0.05) {
    throw new Error(`Too many invalid rows: ${failures.length}/${rows.length}`);
  }

  return parsed
    .filter(result => result.success)
    .map(result => result.data);
}

This gives you a clear failure mode. Instead of discovering bad data in a dashboard two weeks later, the pipeline rejects the run immediately.

Async jobs need idempotency and polling limits

Many scraping APIs use an async pattern: submit a job, get an ID, poll until the result is ready. That works well for long-running jobs, but you need to protect yourself from duplicate submissions and endless polling.

A minimal polling loop should have a timeout, backoff, and a stable idempotency key for the input.

import crypto from "node:crypto";

function idempotencyKey(input: unknown) {
  return crypto
    .createHash("sha256")
    .update(JSON.stringify(input))
    .digest("hex");
}

async function pollJob(jobId: string) {
  const deadline = Date.now() + 10 * 60 * 1000;
  let delayMs = 1000;

  while (Date.now() < deadline) {
    const res = await fetch(`https://api.example.com/jobs/${jobId}`);
    const job = await res.json();

    if (job.status === "succeeded") return job.result;
    if (job.status === "failed") throw new Error(job.error ?? "Job failed");

    await new Promise(resolve => setTimeout(resolve, delayMs));
    delayMs = Math.min(delayMs * 1.5, 15000);
  }

  throw new Error(`Timed out waiting for job ${jobId}`);
}

Without these guards, a network retry can submit the same expensive scrape twice, and a stuck job can keep a worker occupied forever.

Authenticated scraping has a different failure profile

Public pages mostly fail because markup changes or bot defenses get stricter. Logged-in scraping adds another set of problems: expired cookies, two-factor prompts, account locks, regional sessions, and captcha challenges.

A common workaround is to log in once, save cookies and local storage, then reuse them across runs. That works until the session expires or the target site binds it to a proxy, user agent, or device fingerprint.

The minimum you should store for authenticated scraping is:

{
  "cookies": [],
  "localStorage": {},
  "userAgent": "Mozilla/5.0 ...",
  "proxyRegion": "us-east",
  "createdAt": "2026-07-27T10:00:00Z",
  "expiresAt": "2026-07-28T10:00:00Z"
}

Do not treat this as a permanent credential. Treat it as a short-lived session artifact. Encrypt it, rotate it, and expect refresh failures.

Wire handles authenticated actions through named identities rather than asking each job to rebuild login state, which avoids a class of failures caused by repeatedly automating the login flow.

The practical comparison

When you evaluate a scraping platform, do not stop at "can it scrape this site?" Most tools can handle the happy path demo.

Ask these questions instead:

Can I estimate the cost before the job starts?
Do failed or partial runs still cost money?
How do I detect valid-looking but wrong data?
Who fixes the scraper when the site changes?
Can I reuse authenticated sessions safely?
What happens if I submit the same job twice?
Does the platform fit batch jobs, continuous pipelines, or both?

If the answers are unclear, wrap the API with your own run ledger, schema validation, idempotency keys, and timeout handling before sending production traffic through it.

Reducing Browser Fingerprint Detection Starts With Consistency

Anakin — Mon, 27 Jul 2026 10:00:01 +0000

A scraper that works locally can still fail in production with a 403, a CAPTCHA loop, or a blank page after login. The proxy is fine. The headers look normal. navigator.webdriver is patched. The problem is often simpler: the browser fingerprint does not describe one believable device.

Modern bot systems do not rely on one signal. They combine canvas rendering, WebGL strings, timezone, locale, screen size, fonts, TLS handshake shape, IP geolocation, request timing, cookies, and storage state. Any one value can look harmless by itself. The mismatch is what gives you away.

Do not randomize everything

A common first attempt is to randomize every visible browser property on every request. That usually makes detection easier.

A real user does not switch from a UK timezone to a US locale to an Indian IP across three page loads in the same session. A real Chrome browser does not send Firefox-like TLS handshakes while claiming a Chrome user agent. A real GPU does not produce a new canvas fingerprint every time toDataURL() runs on the same page.

The goal is not maximum entropy. The goal is a coherent profile.

For one browser session, these values should agree:

Proxy country
Browser timezone
navigator.language
Accept-Language header
User agent
Screen size
WebGL vendor and renderer
TLS fingerprint
Cookies and localStorage

If you rotate, rotate the whole profile between sessions, not individual fields inside a session.

Patch the obvious automation flags, but do not stop there

Stealth plugins help with basic checks. For Puppeteer, that usually means puppeteer-extra-plugin-stealth:

npm install puppeteer puppeteer-extra puppeteer-extra-plugin-stealth

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');

puppeteer.use(StealthPlugin());

const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://example.com');

This patches common signals like navigator.webdriver, missing Chrome runtime objects, and some plugin inconsistencies. That is useful, but it is not a full fingerprint strategy.

The edge case shows up when the target validates signals together. You might hide navigator.webdriver, but still send:

A datacenter IP in Virginia
Asia/Kolkata as the browser timezone
en-GB as the locale
A user agent for Chrome on Windows
A TLS fingerprint that does not match Chrome

That request does not look like a privacy-conscious user. It looks assembled.

Validate geo alignment before you scrape

You can catch a lot of failures before hitting the target page. Run a small check inside the browser context and compare it with the proxy you selected.

async function validateBrowserGeo(page, expected) {
  return page.evaluate((expected) => {
    const actual = {
      timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
      language: navigator.language,
      languages: navigator.languages,
      platform: navigator.platform
    };

    const errors = [];

    if (actual.timezone !== expected.timezone) {
      errors.push(`timezone=${actual.timezone}, expected=${expected.timezone}`);
    }

    if (!actual.language.startsWith(expected.languagePrefix)) {
      errors.push(`language=${actual.language}, expected prefix=${expected.languagePrefix}`);
    }

    return {
      ok: errors.length === 0,
      actual,
      errors
    };
  }, expected);
}

const result = await validateBrowserGeo(page, {
  timezone: 'Europe/London',
  languagePrefix: 'en-GB'
});

if (!result.ok) {
  throw new Error(`Fingerprint geo mismatch: ${result.errors.join('; ')}`);
}

A failure here is better than a mysterious 403 later:

Fingerprint geo mismatch: timezone=America/New_York, expected=Europe/London; language=en-US, expected prefix=en-GB

If your workflow needs browser execution, proxy country selection, and session state to stay aligned across many requests, Wire treats those as parts of the same extraction profile rather than separate settings you have to keep in sync yourself.

Canvas and WebGL spoofing need stable noise

Canvas fingerprinting hashes pixel-level rendering differences. WebGL exposes values like the unmasked renderer and vendor. Spoofing these can help, but unstable spoofing creates a different problem.

This kind of patch injects small canvas noise:

await page.addInitScript(() => {
  const original = HTMLCanvasElement.prototype.toDataURL;

  HTMLCanvasElement.prototype.toDataURL = function (...args) {
    const ctx = this.getContext('2d');

    if (ctx && this.width > 0 && this.height > 0) {
      const image = ctx.getImageData(0, 0, this.width, this.height);

      for (let i = 0; i < image.data.length; i += 4) {
        image.data[i] += 1;     // red
        image.data[i + 1] += 0; // green
        image.data[i + 2] -= 1; // blue
      }

      ctx.putImageData(image, 0, 0);
    }

    return original.apply(this, args);
  };
});

The important part is not the exact noise. The important part is stability. If the same session produces a different canvas hash on each call, the noise itself becomes a fingerprint.

A better approach is to generate deterministic noise from a session seed and keep it stable for that site and session. Rotate it only when you rotate the full browser profile.

TLS fingerprints are outside JavaScript

JavaScript patches cannot fix TLS fingerprints. JA3 and JA4-style fingerprints come from cipher suite ordering and TLS extensions during the handshake. The server can inspect them before your HTTP headers arrive.

This matters when you use browser automation through a proxy or custom HTTP client. If your headers claim Chrome but your TLS handshake looks like a generic Node.js client, some systems will block before your page script runs.

You usually do not want to invent TLS fingerprints. Use a browser or client that impersonates real browser handshakes, then keep the user agent and browser version consistent with that choice.

Session state is part of the fingerprint

Authenticated scraping adds another layer. Login flows create cookies, localStorage entries, CSRF tokens, feature flags, and sometimes device identifiers. If you log in on every request, clear storage between pages, or reuse cookies with a different fingerprint, you create patterns that normal browsing does not produce.

A practical session flow looks more like this:

Open the home page.
Navigate to login.
Submit credentials.
Visit one or two normal intermediate pages.
Store cookies and localStorage.
Reuse the same browser profile for later target pages.

When extraction requires warmed sessions, Wire can keep browser-backed jobs tied to persisted session state so retries do not accidentally look like new devices.

What to test before calling it fixed

Run the same workflow at least 10 times and record the fingerprint values you care about:

IP country
Timezone
Locale
User agent
WebGL vendor and renderer
Canvas hash
TLS profile, if your tooling exposes it
Status code
CAPTCHA presence
Redirects to bot-check pages

You are looking for two things: consistency inside a session and believable variation between sessions.

Also be honest about the boundary. Fingerprint evasion can violate a site's terms of service, and authenticated scraping has different legal and contractual risks than collecting public pages. Check that before you build around it.

A good next step is to add a fingerprint validation script to CI or to your scraping job startup path. Fail fast when timezone, locale, proxy country, or storage state drift instead of debugging another unexplained 403 in production.

Oxylabs Alternatives: When AI Workflows Need Browser Sessions, Not Just Proxies

Anakin — Fri, 17 Jul 2026 10:00:01 +0000

You start with a scraper that works fine on public pages. Then someone asks the agent to log in, open a dashboard, submit a form, and extract the result after an AJAX update. Suddenly your proxy rotation setup starts failing in odd ways: redirects back to /login, 403 after the second page, invalid CSRF tokens, or sessions that work locally but die in production.

The issue usually is not the proxy itself. It is that modern authenticated sites bind state to more than a cookie. They may correlate IP address, browser fingerprint, timezone, locale, localStorage, sessionStorage, and navigation history. If you rotate IPs on every request, you solve one problem and create another.

Proxy rotation and browser sessions solve different problems

Proxy-heavy tools such as Oxylabs and Bright Data are a good fit when you need to fetch public pages at scale. Product listings, public directories, news archives, SERPs, and region-locked static content are common examples. Rotating residential IPs helps distribute traffic and avoid rate limits.

That model gets awkward when the workflow depends on continuity.

A simplified proxy-rotation scrape looks like this:

const urls = [
  'https://example.com/account',
  'https://example.com/account/billing',
  'https://example.com/account/export'
]

for (const url of urls) {
  const res = await fetch('https://proxy-api.example/fetch', {
    method: 'POST',
    body: JSON.stringify({
      url,
      country: 'US',
      rotateIp: true
    })
  })

  console.log(url, res.status)
}

This can fail even if the first request succeeds. The second request may come from a different IP with no matching browser context. The site sees a cookie from one session, an IP from another, and a fingerprint that may not match either. Common symptoms are 401 Unauthorized, 403 Forbidden, a redirect to login, or a page that renders but contains no user-specific data.

Browser-native platforms take the opposite approach. They keep a browser context alive, including cookies, localStorage, headers, viewport, and often timezone or locale. Browserbase, Browserless, Steel, Firecrawl, and similar tools live in this category, though they expose different levels of control.

Wire is an example of the browser-session side of this tradeoff, where authenticated extraction is modeled around persistent sessions and job state rather than one isolated proxy request per URL.

The decision point is state, not vendor category

A simple rule works most of the time:

Use proxy rotation when the target data is public and each URL can be fetched independently.
Use browser sessions when the workflow logs in, submits forms, follows multi-page state, or depends on JavaScript-rendered data.

The expensive mistake is buying a larger proxy pool to fix a session problem. More IPs do not help if the site expects the same browser identity across five steps.

Here is the shape of a browser-session workflow:

import { chromium } from 'playwright'

const browser = await chromium.launch()
const context = await browser.newContext({
  locale: 'en-US',
  timezoneId: 'America/New_York'
})

const page = await context.newPage()

await page.goto('https://example.com/login')
await page.fill('input[name=email]', process.env.EMAIL!)
await page.fill('input[name=password]', process.env.PASSWORD!)
await page.click('button[type=submit]')

await page.waitForURL('**/dashboard')

await page.goto('https://example.com/account/export')
await page.click('text=Generate export')
await page.waitForSelector('[data-status="complete"]')

const downloadUrl = await page.getAttribute('a[data-export]', 'href')
console.log(downloadUrl)

await context.storageState({ path: 'session.json' })
await browser.close()

The important part is not Playwright itself. It is the persisted context. If you load session.json on the next run, you can often skip login entirely:

const context = await browser.newContext({
  storageState: 'session.json',
  locale: 'en-US',
  timezoneId: 'America/New_York'
})

This pattern has edges. Sessions expire. Password changes invalidate stored state. Some sites require MFA every time the IP changes. Long-running browser sessions consume more memory and cost more than plain HTTP requests. If you run hundreds of them concurrently, you need queueing, timeouts, cleanup, and retry logic.

Async job APIs fit browser work better than blocking requests

Browser work is slow compared with fetching HTML. A page may need 5 to 30 seconds to load, run JavaScript, pass bot checks, and render data. If your API blocks until everything finishes, your workers spend most of their time waiting.

A better pattern is submit, poll, fetch result:

async function submitJob(url: string) {
  const res = await fetch('https://scraper.example/jobs', {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({
      url,
      mode: 'browser',
      sessionId: 'acct_123',
      country: 'US'
    })
  })

  if (!res.ok) throw new Error(`submit failed: ${res.status}`)
  return res.json() as Promise<{ jobId: string }>
}

async function waitForJob(jobId: string) {
  while (true) {
    const res = await fetch(`https://scraper.example/jobs/${jobId}`)
    const job = await res.json()

    if (job.status === 'succeeded') return job.result
    if (job.status === 'failed') {
      throw new Error(job.error ?? 'scrape failed')
    }

    await new Promise(resolve => setTimeout(resolve, 2000))
  }
}

This lets your application submit many browser tasks without tying up one process per page load. It also gives you a clear place to handle failure states such as captcha_required, login_expired, timeout, or blocked.

Wire uses this kind of job-oriented model for browser-based extraction, which matters when several authenticated scraping tasks need to run in parallel and report failure cleanly.

How the common options map out

Oxylabs and Bright Data make sense when IP diversity is the main requirement. Oxylabs publishes residential proxy pricing around bandwidth tiers, with examples like $300/month for 50GB in some plans. That can be reasonable for high-volume static scraping, but it is not designed around preserving login state across arbitrary browser steps.

Browserbase is closer to managed Chromium infrastructure. It fits teams that want Playwright or Puppeteer control without running their own browser fleet. You still need to think about session storage, concurrency, and cost per browser minute.

Firecrawl is useful when the output you want is clean Markdown or structured text from rendered pages. It is a better fit for RAG ingestion than for workflows that need to click through authenticated application state.

Browserless gives you hosted browser automation with its own query layer and browser APIs. Steel is interesting if you want open-source, self-hostable browser infrastructure and you have the ops capacity to run it.

None of these categories is universally better. Static scraping wants throughput and low cost per page. Authenticated agent workflows want continuity and debuggable state.

A practical way to choose

Before comparing pricing pages, classify one real workflow:

Does every URL stand alone, or does page 3 depend on state from page 1?
Does the site bind login to IP, device, locale, or MFA?
Do you need raw browser control, or only extracted text and fields?
What is the acceptable failure mode: retry, manual re-auth, partial result, or hard fail?
How many concurrent browser sessions do you actually need?

Then build a small test that runs the same workflow three ways: plain HTTP, HTTP through rotating proxies, and a persistent browser context. Log status codes, redirects, final URL, extracted field counts, runtime, and cost per successful result.

If plain HTTP works, keep it. If proxy rotation works, use it. If either fails after login, stop tuning proxies and test persistent browser sessions instead.

When browser automation is the wrong layer for agent integrations

Anakin — Thu, 16 Jul 2026 10:00:01 +0000

If you have ever wired an agent to a site without an official API, you have probably reached for Playwright or Puppeteer first. It works in the demo. Then production traffic starts, Chromium eats memory, modal dialogs block flows, a CSS selector changes, and your “API” starts failing because somebody moved a button.

Use the browser to learn the protocol, not to run the workflow

Most modern web apps are API clients. The frontend renders forms and tables, but the useful work usually happens through HTTP calls to private endpoints.

For example, submitting a shipment lookup form might produce a request like this in DevTools:

POST https://carrier.example.com/api/shipments/track
content-type: application/json
x-csrf-token: 4ddf9...
cookie: session=eyJ...

{
  "trackingNumber": "1Z999AA10123456784"
}

If your automation clicks through the UI every time, your runtime path includes page load, JavaScript execution, layout, selectors, screenshots, retries, and whatever experiments the frontend team shipped that week.

If you call the underlying request directly, your runtime path becomes a normal HTTP integration:

import fetch from "node-fetch";

async function trackShipment({ trackingNumber, sessionCookie, csrfToken }) {
  const res = await fetch("https://carrier.example.com/api/shipments/track", {
    method: "POST",
    headers: {
      "content-type": "application/json",
      "x-csrf-token": csrfToken,
      "cookie": `session=${sessionCookie}`,
      "user-agent": "Mozilla/5.0"
    },
    body: JSON.stringify({ trackingNumber })
  });

  if (!res.ok) {
    const body = await res.text();
    throw new Error(`tracking request failed: ${res.status} ${body}`);
  }

  return res.json();
}

That is easier to queue, retry, observe, and test than a headless browser. It is also much cheaper under load because you are not starting a browser process per action.

The tradeoff is that you now own an undocumented API integration.

What actually breaks

Network-level automation does not remove fragility. It moves the fragility to a layer developers can inspect.

Common failure modes look like this:

401 Unauthorized

Your session expired, the site rotated auth cookies, or login now requires a fresh 2FA challenge.

403 Forbidden

The request is missing a CSRF token, origin header, device fingerprint, or some other value the frontend normally supplies.

422 Unprocessable Entity

The endpoint still exists, but the frontend changed the payload shape. Maybe trackingNumber became tracking_number, or the request now needs a hidden accountId fetched from an earlier call.

200 OK with an empty result

This one is worse. Your code did not fail loudly, but the dependency chain changed and you are sending a syntactically valid request with the wrong context.

This is why the interesting work is not “copy request as cURL.” The interesting work is discovering the dependency graph behind the action.

A form submit often depends on earlier calls:

GET /login                         -> sets session cookie
GET /dashboard                     -> returns account id
GET /api/config                    -> returns region and feature flags
GET /api/csrf                      -> returns csrf token
POST /api/shipments/track          -> performs the action

If you replay only the final request, it works until one of those upstream values expires or changes.

Capture the dependency chain explicitly

A practical network-level integration should model each upstream dependency as code, not as a magic blob copied from DevTools.

async function createCarrierClient(credentials) {
  const session = await login(credentials);
  const account = await getAccount(session);
  const config = await getConfig(session);
  const csrfToken = await getCsrfToken(session);

  return {
    async track(trackingNumber) {
      return trackShipment({
        trackingNumber,
        sessionCookie: session.cookie,
        csrfToken,
        accountId: account.id,
        region: config.region
      });
    }
  };
}

This structure gives you places to add retries, cache short-lived values, refresh auth, and log which dependency failed.

It also gives you better tests. You can record fixtures for each dependency and detect when the final request shape changes.

test("track payload matches carrier contract", () => {
  const payload = buildTrackPayload({
    trackingNumber: "1Z999AA10123456784",
    accountId: "acct_123",
    region: "us-east"
  });

  expect(payload).toEqual({
    trackingNumber: "1Z999AA10123456784",
    accountId: "acct_123",
    region: "us-east"
  });
});

For catalog-based integrations, Wire is one example of exposing these network-level workflows as stable action IDs, so application code does not have to store captured request chains directly.

When a browser is still the right tool

Do not turn every website into private HTTP calls by default.

A browser is often the better choice when:

The workflow depends heavily on client-side crypto or WebAuthn
The site streams state through WebSockets with complex client behavior
The request format changes constantly and volume is low
You need visual confirmation, screenshots, or PDF rendering
Legal or compliance rules require using the UI as presented

There is also a maintenance question. If you call undocumented endpoints, someone must watch for drift. That can be your team, a vendor, or a managed integration service, but it cannot be nobody.

Credential handling matters too. Passing raw passwords or session cookies through job payloads creates logs and queues full of secrets. Use a vault-backed identity reference instead of embedding credentials in each request. Wire handles this with named identities, which is the same pattern you would want in a homegrown system: the job receives an identity ID, and the execution layer resolves credentials server-side.

The decision rule I use

For production agent workflows, I usually split the problem like this:

Use official APIs when they exist and cover the workflow
Use direct HTTP replay when the web app has stable private endpoints and volume matters
Use browser automation when the UI itself is part of the workflow or the protocol is too expensive to reverse engineer
Avoid the integration if the site actively blocks automation and you do not have permission to proceed

The key point is that Playwright should not be your default runtime just because it is the fastest way to get a demo working.

Open DevTools, perform the workflow once, and inspect the Network tab. If the useful work is a small number of JSON requests, prototype the HTTP path before you build a browser farm. Then write down the dependency chain, failure modes, auth refresh behavior, and the owner for maintenance before calling it production.

Diagnosing Cloudflare Blocks Before Changing Your Scraper

Anakin — Tue, 14 Jul 2026 10:00:01 +0000

A scraper fails, someone swaps the User-Agent, someone else adds a proxy, then the job starts passing locally but fails again in CI. That usually happens because Cloudflare did not block “scraping” as one thing. It evaluated several signals, and each failure needs a different fix.

This is about authorized automation: your own sites, customer-approved workflows, testing, monitoring, data access you are allowed to perform. If you do not have permission to automate against a site, changing fingerprints or rotating IPs does not make it okay.

Start with the failure you actually see

Cloudflare failures often get collapsed into “403”, but the page body matters.

Common cases:

Error 1020: usually an access denied page from a Cloudflare rule or bot score decision. The HTTP status may still be 403, so inspect the HTML.
403 without a 1020 page: often IP reputation, firewall rules, geo restrictions, or an auth problem.
429: rate limit exhaustion. Slowing down can help here, but it will not fix a fingerprint problem.
Endless Just a moment... page: your client did not complete the browser-side challenge.
CAPTCHA or Turnstile loop: Cloudflare still considers the session borderline after earlier checks.

Add classification before you add workarounds. Even a basic classifier saves time:

import time
import requests

CLOUDFLARE_MARKERS = {
    "1020": "cloudflare_access_denied",
    "Just a moment": "cloudflare_js_challenge",
    "cf-turnstile": "cloudflare_turnstile",
    "cf-error-code": "cloudflare_error_page",
}

def classify_response(resp: requests.Response) -> str:
    body = resp.text[:5000]

    if resp.status_code == 429:
        return "rate_limited"

    for marker, label in CLOUDFLARE_MARKERS.items():
        if marker in body:
            return label

    if resp.status_code == 403:
        return "forbidden_unknown"

    return "ok" if resp.ok else f"http_{resp.status_code}"


def get_with_backoff(url: str, max_attempts=4):
    for attempt in range(max_attempts):
        resp = requests.get(url, timeout=20)
        reason = classify_response(resp)

        print(resp.status_code, reason)

        if reason == "ok":
            return resp

        if reason == "rate_limited":
            retry_after = resp.headers.get("Retry-After")
            sleep_for = int(retry_after) if retry_after else 2 ** attempt
            time.sleep(sleep_for)
            continue

        # Retrying a 1020 or JS challenge with the same client usually repeats the same failure.
        raise RuntimeError(f"Blocked by {reason}")

    raise RuntimeError("Exceeded retries")

The important bit is the branch that does not retry 1020 forever. If the same HTTP client, same IP, and same timing produced a bot decision, repeating it ten times mostly creates more bad reputation.

Match the fix to the detection layer

Cloudflare can evaluate traffic before your application code ever sees it.

At a high level, you are dealing with three layers:

Network and IP reputation: cloud provider ranges, abused proxy pools, and unusual geography can get blocked early.
Protocol fingerprinting: TLS and HTTP/2 details can reveal that the client is requests, curl, or a headless browser wrapper rather than a normal browser.
Browser execution: JavaScript checks can inspect canvas, WebGL, storage, timing, events, and other browser behavior.

A proxy only changes the network source. It does not make requests execute JavaScript. A headless browser can execute JavaScript, but it will not fix a rate limit if you run 100 sessions in parallel from the same account. A CAPTCHA solver might finish a challenge, but it will not help if your session gets blocked again on the next navigation.

If your scraper has become a queue of browser jobs with blocked-page classification, retry state, and extraction output, Wire fits that job-oriented model better than treating every page as a raw HTTP request.

For 403s tied to IP reputation, residential or ISP routing may help, but it adds cost, latency, and operational risk. You also need sticky sessions when login state matters. Rotating IPs per request can break cookies, CSRF tokens, and fraud systems that expect a user to stay on one network path during a session.

Use browser sessions when the page requires browser behavior

If the response body shows Just a moment..., Turnstile markup, or a 1020 page after initial load, a plain HTTP client is probably the wrong tool. Use Playwright or Puppeteer for the parts of the workflow that genuinely need a browser.

Keep session state separate from credentials. Log in once, save cookies and local storage, then reuse that browser state for later runs:

// login.js
import { chromium } from "playwright";

const browser = await chromium.launch({ headless: false });
const context = await browser.newContext();
const page = await context.newPage();

await page.goto("https://example.com/login");

// Complete login manually or through your approved test credentials.
await page.waitForURL("**/dashboard", { timeout: 120000 });

await context.storageState({ path: "auth-state.json" });
await browser.close();

// scrape-authenticated.js
import { chromium } from "playwright";

const browser = await chromium.launch({ headless: true });
const context = await browser.newContext({ storageState: "auth-state.json" });
const page = await context.newPage();

await page.goto("https://example.com/dashboard", { waitUntil: "domcontentloaded" });
await page.waitForLoadState("networkidle");

const title = await page.locator("h1").textContent();
console.log(title);

await browser.close();

This pattern avoids storing passwords in your automation database. You store session artifacts instead. Treat that file like a secret: encrypt it, scope it per user or tenant, and expire it deliberately.

Browser automation costs more than HTTP scraping. It uses more memory, takes seconds instead of milliseconds, and fails in new ways: missing fonts, different viewport behavior, stale session state, and slow third-party scripts. Use it where the site requires it, not as the default for every URL.

Handle rate limits as a product constraint, not a retry loop

A 429 is the one case where slowing down is the correct first response. Respect Retry-After when it exists. If it does not, use exponential backoff with jitter and cap concurrency per account, per IP, and per target host.

Bad scaling pattern:

100 URLs -> 100 parallel browser sessions -> same account -> same origin -> 429 or account lock

Better pattern:

warm session -> process small batch sequentially -> increase concurrency slowly -> record block rate

Measure these separately:

success rate by target route
median and p95 page load time
number of 1020 pages
number of 429 responses
CAPTCHA or Turnstile frequency
retries per successful extraction

That data tells you whether you have a rate problem, a browser challenge problem, or a session credibility problem.

For your next debugging pass, add response classification and per-layer metrics before changing infrastructure. If 429 dominates, tune concurrency. If 1020 dominates, inspect browser execution. If plain 403s dominate, look at auth, geography, and IP reputation first.

Browser automation is not always the right layer for data extraction

Anakin — Mon, 13 Jul 2026 10:00:01 +0000

Your agent needs a product price, a listing count, or a search result. The first version opens Chromium, loads the page, waits for JavaScript, takes a DOM snapshot, and asks an LLM to find the value. It works in a demo. In production, the queue backs up, selectors drift, and half your retry budget goes to pages that rendered a cookie modal instead of the data.

The layer matters

Browser automation is useful, but it is an expensive default for data extraction.

A modern web page is usually not the source of truth. It is a client for backend APIs. The page loads HTML, runs JavaScript, calls internal endpoints, receives JSON, then renders a UI. If your task is “get the price” or “list the search results”, the browser is often doing work you do not need.

There are three common approaches:

Run a browser and inspect the rendered page.
Run a browser with saved site knowledge, such as selectors, workflows, and known endpoint hints.
Call the underlying network API directly and parse structured data.

Browserbase Browse.sh fits the second category. It gives an agent reusable skills so it does not rediscover the same site on every run. That can reduce token use and make browser workflows more predictable, but Chromium and LLM inference still sit in the runtime path.

For covered sites, Wire exposes maintained network-layer actions so an agent can request structured data without rendering the page first.

What the browser path costs you

A typical browser extraction looks something like this:

import { chromium } from "playwright";

const browser = await chromium.launch();
const page = await browser.newPage();

await page.goto("https://example.com/search?q=mechanical+keyboard", {
  waitUntil: "networkidle",
  timeout: 30000
});

const items = await page.locator("[data-testid='result-card']").evaluateAll(cards =>
  cards.map(card => ({
    title: card.querySelector("h2")?.textContent?.trim(),
    price: card.querySelector(".price")?.textContent?.trim()
  }))
);

await browser.close();
console.log(items);

This fails in familiar ways:

TimeoutError: page.goto: Timeout 30000ms exceeded

or:

Error: strict mode violation: locator('.price') resolved to 2 elements

or worse, it succeeds and returns the wrong thing because a sponsored result matched the same selector.

Adding site knowledge helps. A skill file can say “use this selector”, “avoid this path”, or “the first ID in this response is an offset, not a real listing ID.” That matters. It cuts down repeated exploration and makes the agent less likely to waste tokens reasoning about layout.

It does not remove browser startup time, page weight, bot challenges, modal handling, or the need to keep selectors current. If a site changes its rendered markup, your extraction can still break even if the underlying data contract stayed stable.

What network-layer extraction looks like

The network-layer version skips the rendered surface. Instead of asking “where is the price on the page?”, you ask “which request gave the page this data?”

In DevTools, that usually means:

Open the Network tab.
Filter for fetch or xhr.
Perform the search or page action manually.
Find the JSON response that contains the data.
Reproduce that request with the required headers, cookies, query params, and body.

The runtime shape is closer to this:

curl -X POST https://api.openwire.sh/v1/wire/task \
  -H "X-API-Key: $ANAKIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "action_id": "amazon.search_products",
    "params": {
      "query": "mechanical keyboard",
      "limit": 10
    }
  }'

A queued job response is easier to build around than a browser session:

{
  "job_id": "job_01j5x",
  "status": "queued",
  "poll": "/v1/wire/jobs/job_01j5x"
}

Then the completed result is structured data:

{
  "status": "completed",
  "latency_ms": 178,
  "result": {
    "items": [
      {
        "title": "Keychron K2 Pro",
        "price": 99.99,
        "rating": 4.6,
        "reviews": 2340,
        "asin": "B0BJFDQ5KK"
      }
    ]
  }
}

You can build this yourself for a specific site. The hard parts are request signing, session refresh, rate limits, schema drift, and detecting when an endpoint starts returning HTML, a login redirect, or a bot challenge instead of JSON.

That last failure mode is important. If your parser expects JSON and gets this, you want to fail loudly:

SyntaxError: Unexpected token '<', "<!doctype "... is not valid JSON

Do not catch that and return an empty result set. Empty data and failed extraction are different states.

Auth changes the decision

Authentication often decides which layer you can use.

If the workflow requires a real user moving through a multi-step login, completing a form, or handling an interactive challenge, use a browser. Browser sessions preserve the same state the site expects from a human-operated client.

If the workflow is authenticated data retrieval, you may not need to expose raw credentials to the agent at all. For authenticated extraction, Wire uses named identities so code can reference an identity while credential resolution happens outside the agent prompt and response path.

If you build this internally, copy that separation. Store credentials in a vault, pass references through your job system, and make workers fetch short-lived session material at runtime. Do not put passwords, cookies, or bearer tokens in prompts. Also log auth failures distinctly:

401 Unauthorized: session expired
403 Forbidden: request signature invalid
200 OK text/html: login page returned instead of API response

Those three cases need different retries.

A practical rule of thumb

Use network-layer extraction when the output is data, the site has a stable endpoint, and you can validate the response schema.

Use browser automation when the task is visual, interactive, outside your known integrations, or depends on rendered state that is not available from an API. Forms, checkout flows, CAPTCHA-adjacent paths, dashboards with heavy client state, and one-off unknown sites usually belong here.

The mistake is treating the browser as the universal interface. It is an interface, but it is not always the cheapest or most reliable one.

Before adding another browser worker, inspect the Network tab for the action you care about. If the data is already arriving as JSON, prototype the HTTP path, validate the schema, and measure p50 latency, p95 latency, failure rate, and cost per successful extraction against your browser version.

What coding agents need from a real web data layer

Anakin — Thu, 09 Jul 2026 10:00:03 +0000

You ask a coding agent to compare three pricing pages, check a changelog, or pull current compliance dates. It comes back with something plausible, but one source was stale, another page was JavaScript-rendered, and the third was summarized so aggressively that the important caveat disappeared.

That is usually not a model problem. It is a data access problem.

Built-in web tools are lookup tools

Most agent harnesses have some form of web access now. Claude Code has WebSearch and WebFetch. Codex can run with cached, live, or disabled web search. Cursor and other IDE agents expose similar primitives.

These tools help, but they are not the same thing as a web data layer.

Search usually returns titles, URLs, and maybe short snippets. That is enough to decide what to fetch next, but not enough to reason over the source. Fetch tools often return a model-generated summary instead of the original page content. That keeps token usage down, but it also means the agent reasons over another model's interpretation.

The common failure modes are familiar:

Search result: "Pricing | Example SaaS"
Agent conclusion: "The Pro plan costs $49/month"
Actual page: price changed to $59/month behind client-side rendering
Fetched HTML: <div id="root"></div>

Or this:

HTTP 403
<title>Just a moment...</title>
Cloudflare challenge page returned instead of article content

If your agent cannot tell the difference between a real page and a challenge page, it will often continue anyway.

Treat web access as a small data pipeline

A useful web data layer has three separate jobs:

Search and return enough text to rank sources.
Fetch the actual page, including JavaScript-rendered content when needed.
Convert messy page content into typed data the agent can validate.

I like to make that boundary explicit:

type SearchResult = {
  url: string;
  title: string;
  snippet: string;
  date?: string;
};

type FetchResult = {
  url: string;
  status: "completed" | "failed" | "timeout";
  markdown?: string;
  html?: string;
  error?: string;
};

interface WebDataLayer {
  search(query: string, limit: number): Promise<SearchResult[]>;
  fetchPage(url: string, opts?: { renderJs?: boolean }): Promise<FetchResult>;
  extract<T>(input: string, schemaName: string): Promise<T>;
}

This shape matters because it gives the agent concrete states to handle. A failed fetch is not the same as an empty page. A rendered page is not the same as raw HTML. A snippet is not source content.

Anakin's remote MCP connector follows this split with search, scrape, crawl, and structured site actions, but the pattern is useful even if you build the tools yourself.

Async scraping is usually the right default

Fetching a static HTML page can be synchronous. Rendering a SPA, waiting for network calls, rotating proxies, or retrying after a bot challenge should not block an agent request forever.

A better scraper API returns a job ID, then lets the caller poll:

curl -X POST https://example.internal/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com/pricing","renderJs":true}'

# 202 Accepted
# {"jobId":"job_abc123","status":"pending"}

Then your agent-side wrapper handles completion, failure, and timeout explicitly:

async function waitForScrape(jobId: string, timeoutMs = 60_000) {
  const started = Date.now();

  while (Date.now() - started < timeoutMs) {
    const res = await fetch(`https://example.internal/scrape/${jobId}`);
    const job = await res.json();

    if (job.status === "completed") return job;

    if (job.status === "failed") {
      throw new Error(`Scrape failed: ${job.error ?? "unknown error"}`);
    }

    await new Promise(resolve => setTimeout(resolve, 2000));
  }

  throw new Error(`Scrape timed out after ${timeoutMs}ms: ${jobId}`);
}

This is less convenient than a single request, but it gives you better control. You can cap waiting time, log failure reasons, retry selectively, and avoid pretending that a challenge page is valid content.

The tradeoff is latency. Browser rendering can take seconds. For interactive coding sessions, you may want a fast path for normal pages and an async rendered path only when the raw fetch looks wrong.

A simple heuristic catches many misses:

function probablyNeedsBrowser(html: string) {
  return html.includes('<div id="root"></div>') ||
    html.includes('<div id="app"></div>') ||
    html.includes('Just a moment...') ||
    html.length < 1000;
}

It is not perfect. It is better than silently summarizing an empty shell.

Structured extraction should validate output

Agents are good at reading pages, but you should not let them invent data contracts on every run. If you need product names, prices, dates, or availability, define the shape and validate it.

import { z } from "zod";

const PricingPlan = z.object({
  name: z.string(),
  monthlyPriceUsd: z.number().nullable(),
  limits: z.array(z.string()),
  sourceUrl: z.string().url()
});

const PricingPlans = z.array(PricingPlan);

const raw = await web.extract<unknown>(markdown, "pricing_plans");
const plans = PricingPlans.parse(raw);

If parsing fails, you have a real error:

ZodError: Expected number, received string at [0].monthlyPriceUsd

That is much easier to debug than a paragraph saying "the pricing appears to be...".

For sites without APIs, selector-based scrapers often break when teams rename classes or move content into a new component. Wire addresses that specific case with pre-built read actions that return typed JSON for supported sites instead of asking the agent to infer structure from HTML.

Coverage is the obvious tradeoff. A catalog action is useful when it exists and stays maintained. A generic scraper still needs to handle everything else.

Expose the tools through MCP if your agent supports it

MCP is a practical way to give different agents the same tool boundary. The exact server can be your own service, a hosted connector, or a thin wrapper around internal APIs.

For example, a Claude Code setup might look like this:

claude mcp add --transport http webdata https://mcp.example.internal/mcp

Cursor uses JSON config instead:

{
  "mcpServers": {
    "webdata": {
      "url": "https://mcp.example.internal/mcp"
    }
  }
}

The important part is not the client. It is that the agent sees separate tools with clear behavior: search, fetchPage, crawl, and extractStructuredData. Do not hide every operation behind one vague "browse the web" function.

A reasonable starting point

Start small. Add search results with snippets, a page fetcher that can detect obvious bad HTML, and schema validation for one workflow you actually care about. Log the source URL, fetch mode, status, content length, and validation errors.

Then run your agent against five pages that have caused problems before: a static doc page, a SPA, a pricing page, a bot-protected page, and a page with stale indexed content. The gaps will show up quickly, and you can decide whether to add rendering, async jobs, crawling, or site-specific structured actions from there.

Polling Async Web Data Jobs Without Burning Your API Quota

Anakin — Thu, 02 Jul 2026 10:00:01 +0000

Your RAG pipeline looks fine in staging, then someone asks about a product price, policy update, or news item that changed three hours ago. The model answers from yesterday's vector index because the crawler runs nightly. Moving to live web retrieval fixes the freshness problem, but it introduces another one: slow, failure-prone network work now sits in the path of your AI system.

Treat web retrieval as a job, not a request

A synchronous scrape is tempting because it is easy to reason about:

request comes in -> fetch page -> parse page -> answer user

That falls apart when one target site takes 20 seconds, rate limits you, or returns a CAPTCHA page. Your worker sits there waiting, and your upstream timeout decides the result.

The more reliable pattern is:

submit job -> receive job_id -> poll status -> read result

This decouples the caller from the work. The orchestrator can submit several jobs, persist their IDs, and resume later if a process restarts. Most scraping and extraction APIs that do non-trivial work use some version of this shape.

For structured extraction tasks that return job IDs, Wire follows the same submit, poll, retrieve pattern, so the polling loop you write here applies beyond plain HTML scraping.

A typical submission response should look boring:

{
  "job_id": "job_123",
  "status": "pending"
}

The important part is that the API returns quickly. The expensive work happens elsewhere.

Poll slowly enough to matter

The easiest mistake is polling every 500ms because it feels responsive. If most jobs finish in 3 to 15 seconds, those early polls only return this:

{
  "job_id": "job_123",
  "status": "processing"
}

You gain nothing, but you spend rate-limit budget and sometimes billable API calls.

A practical default is:

Wait 3 to 5 seconds before the first poll for normal HTTP scraping.
Wait around 10 seconds before the first poll for browser-rendered pages.
Use exponential backoff after that.
Cap the interval so a long job does not disappear for too long.

Here is a TypeScript implementation I would be comfortable putting behind a queue worker:

class JobTimeoutError extends Error {}
class JobFailedError extends Error {}

type JobStatus<T> =
  | { status: 'pending' | 'processing' }
  | { status: 'completed'; result: T }
  | { status: 'failed'; error?: string };

async function sleep(ms: number) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

async function pollJob<T>(
  jobId: string,
  getStatus: (jobId: string) => Promise<JobStatus<T>>,
  options = {
    initialDelayMs: 5000,
    maxDelayMs: 60000,
    maxAttempts: 10
  }
): Promise<T> {
  let delay = options.initialDelayMs;

  for (let attempt = 0; attempt < options.maxAttempts; attempt++) {
    await sleep(delay);

    const job = await getStatus(jobId);

    if (job.status === 'completed') {
      return job.result;
    }

    if (job.status === 'failed') {
      throw new JobFailedError(job.error ?? `Job ${jobId} failed`);
    }

    delay = Math.min(delay * 2, options.maxDelayMs);
  }

  throw new JobTimeoutError(`Job ${jobId} did not finish after ${options.maxAttempts} polls`);
}

With a 5 second initial delay and a 60 second cap, the loop polls at roughly 5s, 10s, 20s, 40s, then 60s intervals. That is usually enough for web extraction jobs without hammering the status endpoint.

Separate retryable failures from permanent ones

Retries are not a moral good. They help when the failure is transient. They waste money and hide bugs when the request is wrong.

I usually classify failures like this:

Symptom	Likely cause	Action
`502 Bad Gateway`, `503 Service Unavailable`	Upstream or provider issue	Retry with backoff
Network timeout	Temporary network failure	Retry with backoff
`429 Too Many Requests`	Rate limit	Wait for reset, then retry
`400 Bad Request`	Malformed payload	Abort and fix the caller
`401 Unauthorized`	Bad or expired credentials	Abort and alert
`403 Forbidden`	Blocked, missing auth, or wrong region	Usually abort, sometimes retry with different routing

The failure mode matters. If your payload is invalid JSON, five retries produce five invalid requests. If credentials expired, retries just delay the alert that someone needs to refresh the session.

Add a circuit breaker per target

Even good retry logic can hurt you when a whole domain changes behavior. Maybe the site moved prices behind client-side rendering. Maybe it started returning login pages. Maybe your parser now throws Unexpected token < in JSON at position 0 because it expected an API response and got HTML.

A small circuit breaker keeps that from taking down the rest of the pipeline:

type BreakerState = {
  failures: number;
  openedUntil?: number;
};

const breakers = new Map<string, BreakerState>();

function canFetch(domain: string, now = Date.now()) {
  const state = breakers.get(domain);
  return !state?.openedUntil || state.openedUntil <= now;
}

function recordFailure(domain: string, now = Date.now()) {
  const state = breakers.get(domain) ?? { failures: 0 };
  state.failures += 1;

  if (state.failures >= 3) {
    state.openedUntil = now + 5 * 60 * 1000;
  }

  breakers.set(domain, state);
}

function recordSuccess(domain: string) {
  breakers.delete(domain);
}

Start with 3 consecutive failures and a 5 minute cool-off. Tune it from production data. The goal is not to avoid every failed request. The goal is to stop one broken target from consuming your whole worker pool.

Store job IDs like production data

If a job matters, persist its ID with the original request, target URL, tenant, attempt count, and deadline. Logs are not enough. When a downstream workflow times out, you want to answer basic questions:

Did we submit the job?
Is it still processing?
Did it fail permanently?
Which tenant or source caused the retry storm?

A simple table is often enough:

create table web_fetch_jobs (
  id text primary key,
  tenant_id text not null,
  target_url text not null,
  status text not null,
  attempts integer not null default 0,
  deadline_at timestamptz not null,
  created_at timestamptz not null default now(),
  updated_at timestamptz not null default now()
);

Once you have this, you can run polling from a queue, recover after deploys, and inspect stuck jobs without guessing.

The practical next step is to audit any live retrieval path that still blocks on a single HTTP request. Replace it with job submission, persisted IDs, bounded polling, and explicit failure classification.

The full breakdown is here if you want the complete picture.

Before you automate a browser, check the network tab

Anakin — Mon, 29 Jun 2026 10:00:01 +0000

You need data from a website, so the first instinct is often Playwright, Puppeteer, Selenium, or an AI browser agent. That works, but it is easy to miss something important: the browser probably did not create the data you want. It fetched it from an internal API and rendered it.

Open DevTools on a product page, job board, travel site, or finance dashboard. Filter the Network tab to fetch or xhr. Reload the page. A lot of the useful stuff is already JSON.

The browser is often the expensive part

A browser automation script has real costs:

Chromium has to start or be kept warm
The page has to load assets you may not care about
Selectors break when the UI changes
Timeouts happen when ads, modals, or lazy loading behave differently
If an LLM agent drives the browser, every step may involve screenshots, DOM snapshots, and model calls

For example, this Playwright script can extract product data from rendered HTML:

import { chromium } from "playwright";

const browser = await chromium.launch();
const page = await browser.newPage();

await page.goto("https://example.com/products/123", {
  waitUntil: "networkidle",
  timeout: 30000,
});

const price = await page.locator("[data-testid='price']").innerText();
const title = await page.locator("h1").innerText();

console.log({ title, price });
await browser.close();

This is fine until the site renames data-testid, renders the price after a delayed client-side request, or shows a cookie modal that covers the page. The failure usually looks boring:

locator.innerText: Timeout 30000ms exceeded.
Call log:
  - waiting for locator('[data-testid="price"]')

That error tells you the selector failed, not whether the data was unavailable. The JSON endpoint may still have returned the price correctly.

Start by finding the data source

Before writing browser automation, capture the request that populated the page.

In Chrome DevTools:

Open Network
Filter by fetch or xhr
Reload the page
Click likely requests
Check Response and Preview for structured JSON
Right click the request and copy as cURL

You might end up with something like this:

curl 'https://www.example.com/api/products/123?currency=USD' \
  -H 'accept: application/json' \
  -H 'user-agent: Mozilla/5.0' \
  -H 'x-client-version: web-2026.06.1'

If that works outside the browser, you can usually replace a browser job with a normal HTTP client:

const res = await fetch("https://www.example.com/api/products/123?currency=USD", {
  headers: {
    accept: "application/json",
    "user-agent": "Mozilla/5.0",
    "x-client-version": "web-2026.06.1",
  },
});

if (!res.ok) {
  throw new Error(`HTTP ${res.status}: ${await res.text()}`);
}

const product = await res.json();
console.log(product.price, product.availability);

That is the clean path: less infrastructure, lower latency, fewer moving parts, and easier retries.

For teams that want maintained access to these private website APIs instead of managing request signatures themselves, Wire exposes cataloged site actions as normal REST calls with structured JSON responses.

Where direct HTTP gets messy

The copied cURL request is often not enough in production.

Many sites include values that change per session or per request:

CSRF tokens
signed query parameters
device or browser fingerprints
short-lived session cookies
nonces that prevent replay
GraphQL operation hashes

The failure mode is usually a 401, 403, or a JSON response that looks valid but contains an application-level denial:

{
  "error": "invalid_signature",
  "message": "request signature expired"
}

This is where a direct network approach stops being just “copy as cURL”. You need to understand how the frontend generates those headers and tokens. Sometimes that is straightforward. Sometimes the signing code changes weekly.

For supported sites, Wire maintains stable action identifiers while handling endpoint changes, signatures, and authenticated session details behind the API boundary.

If you build this yourself, treat the integration like any other external dependency. Add contract tests against the response shape. Alert on 403 spikes. Store HAR files for debugging. Do not assume a private endpoint is stable just because it returned JSON today.

When a browser is still the right tool

Direct HTTP is not always better. It is better when the data already exists as a request you can reproduce reliably.

Use browser automation when the task depends on the visual state of the page:

multi-page forms with conditional fields
portals that require interactive navigation
flows where validation messages affect the next action
sites that render data only after complex client-side state changes
tasks where a human would say “click the second matching result”

This is where tools like Skyvern make sense. Skyvern runs Chromium, observes screenshots and DOM, asks an LLM what to do, and executes actions through Playwright. Its cached runs can replay generated Playwright scripts, which removes the LLM from successful repeat paths, but it still runs a browser.

That distinction matters. A cached browser workflow may avoid model latency, but it still pays for page load, browser memory, navigation, and UI fragility. A live AI browser agent can adapt to UI changes, but it may take many seconds per step and can fail in ways that are harder to reproduce than a bad HTTP response.

A practical decision rule

Use this order:

Check the Network tab first
If the data is available as JSON, try to reproduce the request with curl
If the request needs tokens or signatures, decide whether maintaining that logic is worth it
If the workflow is visual, stateful, or form-heavy, use browser automation
If volume is high, avoid browsers unless there is no reliable network-layer option

The next time you start a scraping or automation task, spend 10 minutes capturing the HAR file before opening Playwright. If you can turn the target action into one authenticated HTTP request with a testable JSON schema, do that first. If you cannot, then reach for the browser with a clearer reason.

How to evaluate a search API for AI agents without confusing search with extraction

Anakin — Wed, 24 Jun 2026 10:00:01 +0000

A lot of AI search integrations look fine in a demo and fall apart in production for a boring reason: search results are not documents. Most search APIs return titles, URLs, snippets, and maybe relevance scores. If your RAG pipeline, research agent, or monitoring job needs full page content, you need a second step. If your agent needs to log in, click through a web app, or act on a platform, you need more than search.

That distinction matters when comparing tools like Anakin and Tavily. Tavily is a focused search and research API. Anakin includes search, extraction, crawling, mapping, browser automation, persistent browser sessions, and platform endpoints. The better choice depends less on who has the nicer search endpoint and more on what your workflow becomes after the first query.

Search results are usually just pointers

A basic search response is useful for ranking candidate sources, not for building a complete context window.

You typically get something like this:

{
  "results": [
    {
      "title": "Pinecone vs Weaviate comparison",
      "url": "https://example.com/vector-db-comparison",
      "snippet": "A comparison of managed and open source vector databases...",
      "score": 0.82
    }
  ]
}

That is not enough if you need to quote the page, chunk it, embed it, or verify claims. You need to fetch and extract the URL afterward.

A safer pipeline looks like this:

async function research(query) {
  const searchResults = await searchApi.search({
    query,
    limit: 10
  });

  const pages = [];

  for (const result of searchResults.results) {
    try {
      const page = await extractionApi.extract({ url: result.url });

      if (!page.text || page.text.length < 500) {
        console.warn("weak extraction", result.url);
        continue;
      }

      pages.push({
        url: result.url,
        title: result.title,
        text: page.text
      });
    } catch (err) {
      console.warn("extract failed", result.url, err.message);
    }
  }

  return pages;
}

The important part is not the exact SDK. It is the control flow: search first, extract second, handle partial failure. Extraction fails in normal cases: JavaScript-heavy pages, bot checks, paywalls, broken HTML, redirects, consent screens, and pages that return a 200 status with almost no meaningful content.

Test the workflow you actually need

If your product only needs public web search, Tavily is a reasonable fit. It has a simple API, relevance scoring, research endpoints, and public features around PII filtering and prompt injection protection. Those request-level safety features matter if your agent feeds search output directly into an LLM.

If your product starts at search but eventually needs platform interaction, the evaluation changes. A recruiting agent might search the web at first, then need LinkedIn company data. A real estate workflow might search market pages, then need Zillow-style structured records. A sales ops agent might research companies, then submit forms or send messages.

If your search pipeline later has to read or act inside platforms without public APIs, Wire is the kind of platform endpoint layer that changes the tool decision from search quality to workflow coverage.

Without that kind of layer, teams usually add one of three things:

A second vendor for platform-specific data
Browser automation with Playwright or Puppeteer
Internal scrapers that someone now has to maintain

None of those are wrong. They just have costs. Browser automation is flexible, but it brings session handling, CAPTCHA risk, proxy management, and brittle selectors. Platform-specific vendors can be reliable, but each one adds another contract, auth model, data shape, and failure mode.

Cost comparisons need the full call count

Search pricing often looks cheaper than it really is because teams count only the first API call.

For example, using the numbers from the Anakin and Tavily public pricing comparison:

Anakin Scale: $100/month for 120,000 credits
Anakin search: 3 credits per query, so about 40,000 searches
Tavily PAYG: $0.008 per credit
Tavily basic search: 1 credit per query

That puts the rough crossover around 12,500 searches per month. Below that, Tavily’s lower-friction usage model can make sense. At 20,000 searches, Tavily PAYG would be about $160 while Anakin Scale stays at $100. At 40,000 searches, Tavily would be about $320 while Anakin remains $100 for search volume within that plan.

But that is only the search call. If every useful result requires extraction, include those calls too. If one user query produces 10 search results and you extract 5 pages, your cost model should count 6 API operations, not 1.

A simple spreadsheet beats guessing:

monthly_user_queries = 8_000
searches_per_query = 1
urls_extracted_per_query = 5

monthly_search_calls = 8_000
monthly_extract_calls = 40_000
total_billable_operations = 48_000

Also account for retries. If 15 percent of extraction calls fail and you retry once, that adds 6,000 more extraction attempts in this example.

Safety features and compliance are not the same thing

Do not treat security checkboxes as interchangeable.

Request-level protections answer questions like:

Does the API filter PII before returning results?
Does it detect prompt injection in retrieved content?
Can I safely pass search output into an LLM without writing my own filters?

Infrastructure and compliance features answer different questions:

Does the vendor have SOC 2 or ISO certification?
How do they store credentials or browser sessions?
Can procurement approve the vendor?
Is tenant isolation documented?

Tavily’s public PII filtering and prompt injection protection are meaningful if you need safety at the search layer. Anakin’s broader stack matters more if your workflow includes crawling, authenticated browsing, or platform actions. Those are different requirements, not a single winner-takes-all category.

A practical evaluation plan

Before choosing a search API, build a small fixture set from your real workload:

20 normal queries
10 obscure or long-tail queries
10 queries where freshness matters
10 pages that require extraction
5 pages likely to fail because of JavaScript, auth, or bot protection

Run each provider through the same script. Store raw responses. Measure:

How many useful sources came back in the top 10
How often extraction returned enough text
Median and p95 latency
Retry rate
Total calls per completed user task
Whether safety filtering happens in the API or in your code

The decision usually becomes obvious after that. Pick the focused search API if your workflow stays search-and-retrieve and its safety features match your requirements. Pick the broader web automation stack if your agent needs authenticated sessions, platform data, browser execution, or write actions.

The next useful step is to write the test harness before wiring either provider into your app. A weekend of fixture-based testing will tell you more than a feature table.

Network-layer scraping is fast, but the maintenance cost is the real decision

Anakin — Tue, 23 Jun 2026 10:00:01 +0000

If you have ever shipped a Puppeteer or Playwright scraper, you know the usual failure mode: it works locally, then production gets slower, a selector changes, a cookie modal appears, or a page loads differently in headless mode. A lot of teams eventually notice that the browser is not where the useful data comes from. The frontend usually calls a JSON endpoint in the background, and that endpoint is what you actually wanted.

The useful trick: replay the request, not the UI

Open DevTools, go to the Network tab, filter by fetch or xhr, then use the site normally. On many modern sites, the page is just a client around internal HTTP, GraphQL, or RPC calls.

For example, a product search page might call something like this:

curl 'https://example.com/api/search?q=headphones&page=1' \
  -H 'accept: application/json' \
  -H 'user-agent: Mozilla/5.0' \
  -H 'x-client-version: web-2026.06.12'

That request might return structured data directly:

{
  "items": [
    { "id": "p_123", "name": "Wireless headphones", "price": 7999, "in_stock": true }
  ],
  "next_page": 2
}

Calling that endpoint from your backend is usually much faster than rendering a page. You avoid DOM waits, image loading, layout work, and most selector problems.

A minimal implementation might look like this:

async function searchProducts(query, page = 1) {
  const res = await fetch(
    `https://example.com/api/search?q=${encodeURIComponent(query)}&page=${page}`,
    {
      headers: {
        accept: 'application/json',
        'user-agent': 'Mozilla/5.0',
        'x-client-version': 'web-2026.06.12'
      }
    }
  );

  if (!res.ok) {
    throw new Error(`Search failed: ${res.status} ${await res.text()}`);
  }

  const body = await res.json();

  if (!Array.isArray(body.items)) {
    throw new Error(`Unexpected schema: ${JSON.stringify(body).slice(0, 300)}`);
  }

  return body.items.map(item => ({
    id: item.id,
    name: item.name,
    priceCents: item.price,
    inStock: item.in_stock
  }));
}

That pattern is simple, cheap, and often good enough. It is also where the real maintenance problem starts.

The endpoint is not a public API

Internal endpoints change without notice. The site owner does not consider your caller a customer of that API.

Common breakages look like this:

401 Unauthorized
{"error":"missing session"}

or:

403 Forbidden
{"message":"invalid client fingerprint"}

or worse, the request still returns 200 OK, but the response shape changes:

{
  "products": [
    { "sku": "p_123", "displayName": "Wireless headphones" }
  ]
}

Your code does not crash at the HTTP layer. It crashes later because body.items is undefined, or it silently writes incomplete data because price moved somewhere else.

This is the tradeoff with network-layer scraping. You remove browser flakiness, but you take ownership of an undocumented API contract.

Generated endpoints are good for exploration

Tools that generate an API from a URL are useful when you need quick coverage. Give the tool a page and a description, wait a short time, and you get an endpoint that extracts fields from the underlying calls.

That is a good fit for:

one-off research
prototypes
read-only extraction
long-tail sites you may never call again
validating whether the target site exposes useful JSON at all

The downside is operational. If the generated endpoint captured /api/v2/search and the site moves to /api/v3/query, someone has to regenerate or repair it. If the site adds a required header, rotates a token format, or changes pagination, your pipeline owns that breakage unless the provider maintains it for you.

For production systems, maintenance matters more than initial generation speed. Wire is one example of the maintained-catalog approach, where the caller keeps using the same action while the provider handles changes in the target site's internal calls.

That model is less flexible for obscure sites, but it fits recurring workflows better.

Read actions and write actions are not the same problem

Reading from an internal endpoint is relatively safe to reason about. You send query params, receive JSON, validate the schema, and store the result.

Writing is harder. A POST request often carries CSRF tokens, session-bound IDs, idempotency keys, hidden form fields, or state derived from previous requests.

A job application flow might look like this:

GET  /jobs/12345
GET  /api/applications/bootstrap?job_id=12345
POST /api/applications/draft
POST /api/applications/draft/resume
POST /api/applications/submit

If you replay only the final POST, you may get:

409 Conflict
{"error":"draft_not_initialized"}

If you reuse an old token, you may get:

422 Unprocessable Entity
{"error":"csrf_token_expired"}

If you retry a non-idempotent action after a timeout, you may submit twice.

This is why read-only extraction and web action automation should not share the same risk model. Reading a price wrong is bad. Sending the same customer message twice is a different class of bug. For workflows that submit, post, message, bid, or apply, Wire treats read and write actions as cataloged operations rather than leaving each caller to reconstruct the browser flow.

Even if you build this yourself, make write actions explicit in your codebase. Do not hide them behind a generic fetchPageData() helper.

What I would test before depending on network replay

For any target site, I would answer these questions before calling it production-ready:

Does the endpoint require an authenticated session?
How long do session cookies or tokens last?
Does the response schema stay stable across pages, accounts, and regions?
What status code appears when the site blocks or challenges the request?
Can the request be retried safely?
Do you have a fixture test that fails when the response shape changes?

A simple schema check catches a lot:

function assertSearchResponse(body) {
  if (!body || !Array.isArray(body.items)) {
    throw new Error('Expected body.items to be an array');
  }

  for (const item of body.items) {
    if (typeof item.id !== 'string') throw new Error('Missing item.id');
    if (typeof item.name !== 'string') throw new Error('Missing item.name');
    if (typeof item.price !== 'number') throw new Error('Missing item.price');
  }
}

Run that against real responses on a schedule. If the test fails before your customer-facing job fails, you bought yourself time.

The practical decision

Use browser automation when you need to interact with a UI that has no useful network calls, or when visual state matters.

Use direct network replay when the site exposes structured background requests and you can tolerate maintaining an undocumented contract.

Use generated endpoints when speed matters more than long-term stability.

Use maintained integrations when the same action runs repeatedly, failure has business impact, or the workflow writes back to the target site.

A good next step is to pick one flaky browser scraper, inspect its XHR calls, replay the smallest useful request with curl, then add a schema test around the response. That will tell you quickly whether network replay reduces the problem or just moves the complexity somewhere else.