DEV Community: Anakin

Stop scraping the page when the data is already in the network tab

Anakin — Thu, 18 Jun 2026 10:00:04 +0000

You write a scraper with Playwright, wait for the page to load, close the cookie banner, click a filter, and parse a table out of the DOM. Then someone redesigns the page and your selector breaks. The annoying part is that the data probably never lived in the HTML in the first place.

Most modern websites render a UI around structured background requests. The browser loads the shell, runs JavaScript, and calls internal endpoints for prices, availability, inventory, search results, profile data, or whatever the page needs. If you scrape the rendered page, you often process hundreds of kilobytes of layout and tracking code to recover a few kilobytes of JSON.

Look at the network layer before writing browser code

Before reaching for Playwright or Puppeteer, open DevTools and check what the site actually does.

In Chrome:

Open DevTools
Go to the Network tab
Filter by Fetch/XHR
Perform the action manually, such as search, filter, paginate, or change dates
Click the request that returns the data
Inspect the request URL, method, headers, payload, and response

You will often find something like this:

POST /api/search/hotels HTTP/2
content-type: application/json
x-csrf-token: 8f9c...

{
  "checkIn": "2026-03-12",
  "checkOut": "2026-03-15",
  "city": "Berlin",
  "guests": 2
}

And the response is already the thing you wanted:

{
  "results": [
    {
      "id": "hotel_123",
      "name": "Example Hotel",
      "price": 184,
      "currency": "EUR",
      "available": true
    }
  ]
}

At that point, scraping the DOM is extra work. You can reproduce the request directly:

curl 'https://example.com/api/search/hotels' \
  -X POST \
  -H 'content-type: application/json' \
  -H 'x-csrf-token: 8f9c...' \
  --data '{"checkIn":"2026-03-12","checkOut":"2026-03-15","city":"Berlin","guests":2}'

Or from code:

const res = await fetch('https://example.com/api/search/hotels', {
  method: 'POST',
  headers: {
    'content-type': 'application/json',
    'x-csrf-token': process.env.CSRF_TOKEN
  },
  body: JSON.stringify({
    checkIn: '2026-03-12',
    checkOut: '2026-03-15',
    city: 'Berlin',
    guests: 2
  })
});

if (!res.ok) {
  throw new Error(`Search failed: ${res.status} ${await res.text()}`);
}

const data = await res.json();
console.log(data.results.map(h => [h.name, h.price]));

This is the basic pattern: use the browser to discover the request, not to run every request forever.

Wire works in this network-layer space by mapping the background requests a site already uses and exposing stable endpoints around those flows.

Why this matters in production

Browser automation is useful, but it is expensive for data retrieval.

A headless browser has to start a browser process, load HTML, download scripts, execute JavaScript, build a DOM, run layout work, and wait for the app to settle. If you only need the JSON response behind a search form, most of that work is overhead.

The difference shows up quickly:

// Browser-level approach
const page = await browser.newPage();
await page.goto('https://example.com/hotels?q=berlin');
await page.waitForSelector('[data-testid="hotel-card"]');
const prices = await page.$$eval('[data-testid="hotel-card"]', cards =>
  cards.map(card => card.textContent)
);

// Network-level approach
const res = await fetch('https://example.com/api/search/hotels', options);
const { results } = await res.json();
const prices = results.map(h => h.price);

The first version breaks if the frontend team renames data-testid, changes the card layout, lazy-loads the list differently, or inserts an experiment variant. The second version breaks if the internal API changes, auth changes, or the site adds request signing. Neither is magic. The network version just avoids tying your pipeline to visual presentation.

It also gives less junk to downstream systems. If an LLM needs to answer, "Which hotels are available under 200 EUR?", passing it rendered HTML or markdown forces it to ignore navigation, legal text, ads, and duplicated labels. Passing structured JSON lets you validate, filter, and summarize before the model sees anything.

The edge cases are real

Internal endpoints are not public contracts. Treat them as dependencies that can change without notice.

Common failure modes include:

401 or 403 because the request needs a logged-in session
400 because the frontend generated a hidden token you did not include
429 because you ignored rate limits
Empty results because a required header, locale, currency, or experiment flag is missing
A successful 200 response with a changed JSON shape
Requests signed with timestamps or hashes generated in bundled JavaScript

A small amount of defensive code helps:

function parseHotelSearch(payload) {
  if (!payload || !Array.isArray(payload.results)) {
    throw new Error('Unexpected hotel search response shape');
  }

  return payload.results.map(item => {
    if (typeof item.name !== 'string' || typeof item.price !== 'number') {
      throw new Error(`Invalid hotel item: ${JSON.stringify(item)}`);
    }

    return {
      id: item.id,
      name: item.name,
      price: item.price,
      currency: item.currency ?? 'UNKNOWN',
      available: item.available === true
    };
  });
}

You still need monitoring. Log status codes, response sizes, parse failures, and schema changes. If the payload usually contains results and suddenly returns items, you want an alert before a customer notices missing data.

For teams that do not want to own endpoint discovery, request signing, and breakage handling for many third-party sites, Wire is a managed version of this same approach rather than a DOM scraping wrapper.

When a browser is still the right tool

Use browser automation when the browser behavior is the thing you need to test or reproduce.

Good cases for Playwright, Puppeteer, or Selenium:

End-to-end testing user flows
Capturing screenshots or PDFs
Interacting with canvas-heavy or browser-only apps
Debugging frontend behavior
Handling flows where the data is only available after complex client-side state changes
Verifying that the UI actually displays what the API returned

Bad cases:

Polling prices every five minutes
Pulling paginated search results
Checking inventory across many SKUs
Feeding structured records into a data pipeline
Giving an agent live availability data

For those, inspect the network requests first. If the data is already JSON, call that layer directly, validate the response shape, and keep the browser out of the hot path unless you actually need it.

Choosing Between Scraping APIs, Browser Sessions, and Self-Managed Playwright

Anakin — Wed, 17 Jun 2026 10:00:02 +0000

Most teams do not start by designing scraping infrastructure. They start with a script. Then the script becomes a cron job, the cron job becomes a queue, and eventually someone is debugging why Chromium processes are eating all the memory on a worker node. At that point, the question is not “how do we scrape this page?” It is “which parts of this system do we actually want to own?”

The three common options

For continuously updated AI systems, most web data pipelines land in one of three patterns.

Pattern	Best fit	Main cost	Common failure
Async scraping API	Public pages, batch ingestion, RAG refreshes	Per job or per request	Vendor limits, polling complexity
Hosted browser sessions	Authenticated flows, multi-page navigation	Session time	Paying for idle browser minutes
Self-managed Playwright or Puppeteer	High volume, custom browser behavior	Engineering time and infrastructure	Proxy, retry, and browser maintenance

None of these is universally better. The right choice depends on volume, authentication, latency, and how much operational work your team can absorb.

Async APIs: good default for background ingestion

Async APIs work well when you can submit work and collect the result later. That maps cleanly to RAG ingestion, where you usually do not want a user request waiting on a live browser.

The shape is simple:

async function submitUrl(url) {
  const res = await fetch("https://scraper.example.com/jobs", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.SCRAPER_TOKEN}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify({ url, render: true })
  });

  if (!res.ok) throw new Error(`submit failed: ${res.status}`);
  return res.json();
}

You store the returned job ID, poll until completion, validate the result, then update your database or vector store.

The tradeoff is latency. If a job takes 3 to 15 seconds, that is acceptable for background refresh. It is not acceptable inside a chat completion path unless you stream an intermediate answer or tell the user you are fetching fresh data.

Async APIs also introduce rate limits. If the submit endpoint allows 60 requests per minute, this loop will eventually fail:

await Promise.all(urls.map(url => submitUrl(url)));

Use a limiter instead:

import pLimit from "p-limit";

const limit = pLimit(5);

await Promise.all(
  urls.map(url => limit(() => submitUrl(url)))
);

For teams choosing this path, Wire fits the async scraping API category: submit URLs, track job IDs, and consume structured page output without running your own browser pool.

Hosted browser sessions: use them when state matters

Some sources require login, cookies, local storage, or multi-step navigation. A stateless scrape request is a bad fit for that.

Example workflow:

Log in
Navigate to an account page
Open a report
Apply filters
Extract the rendered table

You can do this with a hosted browser session by connecting Playwright to a remote browser endpoint:

import { chromium } from "playwright";

const browser = await chromium.connectOverCDP(process.env.REMOTE_BROWSER_WS);
const page = await browser.newPage();

await page.goto("https://example.com/login");
await page.fill("input[name=email]", process.env.APP_USER);
await page.fill("input[name=password]", process.env.APP_PASSWORD);
await page.click("button[type=submit]");

await page.goto("https://example.com/reports");
await page.click("text=Last 30 days");

const rows = await page.locator("table tbody tr").evaluateAll(nodes =>
  nodes.map(row => row.innerText)
);

await browser.close();

The provider runs the browser. You control the interaction.

This is useful, but the billing model matters. Browser services often charge by session time. A script that waits on slow pages, sleeps between actions, or leaves sessions open after errors can become expensive.

Always close sessions in finally:

let browser;

try {
  browser = await chromium.connectOverCDP(process.env.REMOTE_BROWSER_WS);
  // work
} finally {
  await browser?.close();
}

Hosted browser sessions are a middle ground. You avoid maintaining browser hosts, but you still own workflow logic, selectors, validation, and credentials handling.

Self-managed Playwright: control is not free

Running Playwright yourself gives you the most control. You can install extensions, intercept network calls, tune launch flags, control geographies, and keep browser pools warm.

A basic worker looks harmless:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: true });

for (const url of urls) {
  const page = await browser.newPage();
  try {
    await page.goto(url, { waitUntil: "networkidle", timeout: 30000 });
    const title = await page.title();
    console.log({ url, title });
  } catch (err) {
    console.error(`failed ${url}:`, err.message);
  } finally {
    await page.close();
  }
}

await browser.close();

Production adds the parts this example avoids:

browser pool sizing
queue backpressure
proxy rotation
retry budgets
per-domain rate limits
CAPTCHA and bot detection handling
memory leak monitoring
selector maintenance
alerting when extraction quality drops

The common mistake is comparing API credits to server cost only. That ignores engineering time. If a site redesign breaks ten selectors and one developer spends a day fixing parsers, that cost belongs in the scraping budget.

Self-managed starts to make sense when volume is high enough, the workflows need custom behavior, and the team already has operational capacity. As a rough heuristic, below hundreds of thousands of pages per month, managed options often cost less once maintenance is included. Above that, run the numbers with your actual failure rate and developer time.

A practical decision process

Ask these questions in order.

Does the source require authentication or multi-step state?

If yes, start with hosted browser sessions or self-managed Playwright. Async single-page extraction can work only if you can provide valid session state safely and the page does not need interaction.

Does the user need the data during the current request?

If yes, avoid long async scraping jobs in the hot path. Use cached content, stream progress, or design the product around delayed retrieval. If no, async jobs are usually simpler.

Do you need custom browser behavior?

If you need extensions, custom TLS behavior, request interception, or nonstandard anti-bot handling, self-managed may be justified. If not, owning that stack is probably unnecessary.

Can you tolerate selector maintenance?

If your extraction depends on CSS selectors, treat redesigns as expected events. Add validation and alerts for empty fields. A scraper that returns 200 OK with empty extracted data has still failed.

if (!result.price || !result.name) {
  await alerts.send({
    source: url,
    reason: "extraction returned missing required fields"
  });
}

The short version

Use async APIs for background ingestion from public pages. Use hosted browser sessions when login and navigation state matter. Use self-managed Playwright when you need control badly enough to pay for it with infrastructure and maintenance.

Do not decide from request volume alone. Include failure rates, selector churn, retry behavior, developer time, and whether stale data causes real product problems.

The full breakdown is here if you want the complete picture.

Async Scraping Jobs Are Usually a Better Fit for RAG Ingestion Than Blocking Requests

Anakin — Tue, 16 Jun 2026 10:00:01 +0000

A RAG system that depends on web data usually fails in a boring way: the page changed, your index did not, and the model answers confidently from stale context. The first fix people reach for is often a cron job plus a scraper. That works until page loads take 12 seconds, a target site returns 429s, and your app starts holding open worker threads for data that should have been fetched in the background.

Why synchronous scraping breaks down

A synchronous scraper looks simple:

const html = await fetchPage(url);
const data = extractProduct(html);
await vectorStore.upsert(data);

That code hides a few production problems.

If fetchPage(url) needs a browser, the request might spend several seconds waiting for JavaScript, network calls, cookie banners, or anti-bot checks. If you run this inside an API request handler, the user waits. If you run enough of them in parallel, your workers sit around holding memory and open sockets.

The failure modes are also awkward:

TimeoutError: Navigation timeout of 30000 ms exceeded
HTTPError: Response code 429 (Too Many Requests)
Error: selector ".price" did not match any elements
ProtocolError: Target closed

Each error needs a different response. A timeout might need a retry. A 429 might need backoff or a different proxy. A missing selector might mean the site redesigned its markup, or it might mean the product is unavailable. Treating all of those as try again later creates bad data and noisy queues.

For AI pipelines, the bigger issue is freshness. If your ingestion job runs hourly, the model can be wrong for up to an hour even when every component is technically healthy. For inventory, shipment status, event schedules, job listings, or pricing, that gap matters.

The async job pattern

Async scraping APIs use a submit, poll, retrieve flow. You submit work, get a job ID, and fetch the result later. The important part is not the API shape itself. The important part is that page retrieval no longer blocks your application flow.

A minimal version looks like this:

# Submit work
curl -X POST https://scraper.example.com/jobs \
  -H "Authorization: Bearer $SCRAPER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/products/123","render":true}'

# Response
# { "jobId": "job_8f31", "status": "queued" }

Then poll by ID:

curl https://scraper.example.com/jobs/job_8f31 \
  -H "Authorization: Bearer $SCRAPER_TOKEN"

# Possible response
# { "jobId": "job_8f31", "status": "running" }

# Completed response
# {
#   "jobId": "job_8f31",
#   "status": "completed",
#   "markdown": "# Product 123\n\nPrice: $42",
#   "html": "<html>...</html>",
#   "durationMs": 8420
# }

In application code, keep polling separate from request handling:

async function waitForScrape(jobId, { timeoutMs = 60000 } = {}) {
  const started = Date.now();
  let delay = 2000;

  while (Date.now() - started < timeoutMs) {
    const res = await fetch(`https://scraper.example.com/jobs/${jobId}`, {
      headers: { Authorization: `Bearer ${process.env.SCRAPER_TOKEN}` }
    });

    if (!res.ok) {
      throw new Error(`status poll failed: ${res.status}`);
    }

    const job = await res.json();

    if (job.status === "completed") return job;

    if (job.status === "failed") {
      throw new Error(`scrape failed: ${job.reason || "unknown reason"}`);
    }

    await new Promise(resolve => setTimeout(resolve, delay));
    delay = Math.min(delay * 1.5, 10000);
  }

  throw new Error(`scrape job ${jobId} timed out after ${timeoutMs}ms`);
}

This gives you a clean boundary. Your ingestion worker can submit 500 URLs, store job IDs, and update records as jobs finish. Your user-facing app does not need to keep browsers open or know how proxy retries work.

For this specific async ingestion pattern, Wire exposes web extraction as jobs that return IDs for polling, which matches the way background RAG updates usually need to run.

What you still need to build

Async jobs do not remove all pipeline work. They move the browser and network mess behind an API, but you still need to decide how your system treats results.

At minimum, store job state in your own database:

create table scrape_jobs (
  id text primary key,
  source_url text not null,
  status text not null,
  submitted_at timestamptz not null default now(),
  completed_at timestamptz,
  error text,
  content_hash text
);

The content_hash matters. If the page content did not change, do not re-embed it and write duplicate vectors. A simple hash over normalized Markdown is often enough:

import crypto from "node:crypto";

function hashContent(markdown) {
  return crypto
    .createHash("sha256")
    .update(markdown.replace(/\s+/g, " ").trim())
    .digest("hex");
}

You also need dead-letter handling. If a job fails three times, stop retrying it in the hot path and send it somewhere visible. Otherwise one broken source can consume your queue forever.

if (job.attempts >= 3) {
  await db.deadLetters.insert({
    sourceUrl: job.sourceUrl,
    reason: job.error,
    failedAt: new Date()
  });
  return;
}

Selector drift is a separate problem

Async scraping helps with latency and reliability, but it does not automatically fix extraction logic. If your parser depends on this:

const price = $(".product-price .amount").text();

then a redesign can silently produce an empty string. That is worse than a hard failure because bad data enters the index.

Prefer extraction contracts that validate output before ingestion:

const ProductSchema = z.object({
  name: z.string().min(1),
  price: z.string().min(1),
  availability: z.enum(["in_stock", "out_of_stock", "unknown"])
});

const parsed = ProductSchema.safeParse(extracted);

if (!parsed.success) {
  throw new Error(parsed.error.message);
}

Whether the extraction comes from selectors, readability output, or an AI extraction step, validate it before it reaches your vector store.

Where async jobs fit

Async scraping jobs fit background ingestion, scheduled refreshes, change detection, and batch updates. They do not fit cases where a chat response must include a page fetched milliseconds ago. A typical scrape job may take several seconds, especially with browser rendering.

If the user can wait, fine. If not, use cached content and update it behind the scenes.

The practical pattern is simple: keep scraping out of request handlers, treat each scrape as a job with state, validate the extracted shape, hash content before embedding, and dead-letter repeated failures instead of retrying forever.

The full breakdown is here if you want the complete picture.

Designing credential handling for authenticated browser automation

Anakin — Fri, 12 Jun 2026 10:00:02 +0000

If you build automation that logs into third-party websites, the dangerous shortcut is obvious: save the username and password, then replay them whenever the job runs. It works, until a log line, API response, support tool, database dump, or overbroad internal permission turns that stored password into a real incident.

Treat the password as a login event, not stored state

A safer model is to treat the password as input to one operation: establishing a session. After login succeeds, store the browser session state, not the password.

The basic flow looks like this:

User enters credentials.
Your service uses them to perform an interactive login.
Your service captures cookies and local storage from the browser context.
Your service encrypts that session state and stores it with tenant and user scope.
Your service discards the password.
Future jobs load the encrypted session state instead of asking for the password again.

Here is a simplified Node.js example using Playwright and AES-256-GCM:

import { chromium } from "playwright";
import crypto from "node:crypto";

function encryptJson(value: unknown, key: Buffer) {
  const iv = crypto.randomBytes(12);
  const cipher = crypto.createCipheriv("aes-256-gcm", key, iv);

  const plaintext = Buffer.from(JSON.stringify(value), "utf8");
  const ciphertext = Buffer.concat([
    cipher.update(plaintext),
    cipher.final()
  ]);

  return {
    ciphertext: ciphertext.toString("base64"),
    iv: iv.toString("base64"),
    tag: cipher.getAuthTag().toString("base64")
  };
}

async function connectAccount({
  tenantId,
  userId,
  username,
  password,
  encryptionKey
}: {
  tenantId: string;
  userId: string;
  username: string;
  password: string;
  encryptionKey: Buffer;
}) {
  const browser = await chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();

  try {
    await page.goto("https://example.com/login");
    await page.fill("input[name=email]", username);
    await page.fill("input[name=password]", password);
    await page.click("button[type=submit]");

    await page.waitForURL("**/dashboard", { timeout: 15000 });

    const storageState = await context.storageState();
    const encrypted = encryptJson(storageState, encryptionKey);

    const sessionId = await db.sessions.insert({
      tenantId,
      userId,
      ciphertext: encrypted.ciphertext,
      iv: encrypted.iv,
      tag: encrypted.tag,
      status: "active",
      expiresAt: new Date(Date.now() + 90 * 24 * 60 * 60 * 1000)
    });

    return { sessionId, status: "connected" };
  } finally {
    await browser.close();
  }
}

Do not overstate what this does. Setting a JavaScript variable to undefined does not guarantee memory erasure. The important design choice is that you never write the password to disk, never return it from an API, never log it, and never keep it as reusable application state.

Sessions are credentials too

A browser session can contain cookies, JWTs, refresh tokens, CSRF tokens, and site-specific state in local storage. If someone steals it, they may not need the password.

So session storage needs nearly the same controls as password storage:

CREATE TABLE browser_sessions (
  id uuid PRIMARY KEY,
  tenant_id uuid NOT NULL,
  user_id uuid NOT NULL,
  ciphertext text NOT NULL,
  iv text NOT NULL,
  auth_tag text NOT NULL,
  status text NOT NULL CHECK (status IN ('active', 'revoked', 'expired')),
  created_at timestamptz NOT NULL DEFAULT now(),
  expires_at timestamptz NOT NULL,
  revoked_at timestamptz
);

CREATE INDEX browser_sessions_tenant_id_idx
ON browser_sessions (tenant_id, user_id, status);

Every read should include tenant scope. Do not fetch by session ID alone and then check ownership later. That creates room for mistakes in future call sites.

SELECT *
FROM browser_sessions
WHERE id = $1
  AND tenant_id = $2
  AND user_id = $3
  AND status = 'active'
  AND expires_at > now()
  AND revoked_at IS NULL;

This is also where API shape matters. Return handles and metadata, not secrets:

{
  "sessionId": "6b9f...",
  "status": "active",
  "expiresAt": "2026-03-01T00:00:00Z"
}

Do not return encrypted session blobs to the frontend “because they are encrypted.” If the client does not need the value, the API should not expose it.

For authenticated browser jobs, Wire follows this session-based model: the automation runs against an encrypted browser session rather than keeping a reusable site password around.

Expiry and revocation need first-class behavior

Sessions expire. Sites rotate cookies. Users change passwords. MFA prompts appear. A good system treats these as normal states, not weird failures.

Bad behavior looks like this:

Error: Timeout 30000ms exceeded

That tells the user nothing. Did the page load slowly? Did login expire? Did the selector change? Did MFA block the run?

Better behavior distinguishes the failure:

if (page.url().includes("/login")) {
  throw new SessionExpiredError({
    code: "SESSION_RECONNECT_REQUIRED",
    message: "The saved browser session is no longer valid. Reconnect the account."
  });
}

Revocation should also take effect before the next job starts, not eventually after some background cleanup runs:

UPDATE browser_sessions
SET status = 'revoked', revoked_at = now()
WHERE id = $1
  AND tenant_id = $2
  AND user_id = $3;

Then your job runner must refuse to load revoked sessions. This sounds obvious, but many systems implement revocation only in the UI and forget the worker path.

Logs are the easiest place to leak secrets

Even if the database design is good, logs can undo it. The common mistake is logging request bodies during login debugging:

logger.info({ body: req.body }, "connect account request");

That can put passwords, cookies, or tokens into your observability system. Use structured redaction by default:

import pino from "pino";

export const logger = pino({
  redact: {
    paths: [
      "req.headers.authorization",
      "req.body.password",
      "password",
      "cookies",
      "storageState.cookies",
      "storageState.origins[*].localStorage[*].value"
    ],
    censor: "[REDACTED]"
  }
});

Also check error reporting. Browser automation failures often include screenshots, HTML snippets, network headers, and console logs. Any one of those can contain account data.

Isolate the browser runtime

Do not reuse one persistent browser profile across users or tenants. Create a fresh context for each job, load only the specific session state needed for that job, then tear it down.

const context = await browser.newContext({ storageState });
try {
  await runAutomation(context);
} finally {
  await context.close();
}

This does not solve every problem. The target site can still rate limit you. MFA can still interrupt execution. A compromised automation script can still misuse the access it has during the run. But isolation reduces accidental cross-account state bleed, which is one of the nastier classes of bugs in browser automation systems.

Wire runs authenticated tasks in scoped browser environments, which addresses the specific risk of one customer’s browser state carrying into another job context.

A practical checklist

If you are reviewing or building this kind of system, trace one account connection from end to end and answer these questions:

Is the password ever written to disk?
Can any API response include a cookie, token, or encrypted session blob?
Do all session reads include tenant and user scope?
What does the worker do when the site redirects to login?
Does revocation block future runs immediately?
Are request bodies, screenshots, HTML dumps, and network headers redacted?
Does each job get a fresh browser context?

Start by testing the failure paths. Expire a session manually, revoke one while a job is queued, and search your logs for the test password. That will tell you more than the happy path ever will.

A safer pattern for authenticated automation: vault-backed sessions

Anakin — Thu, 11 Jun 2026 10:00:01 +0000

Authenticated automation usually fails at the same boring place: login. Scraping a public page is easy compared with running a workflow inside a supplier portal, admin dashboard, CRM, or internal tool. At some point your automation needs a password, and the naive answer is to store that password in your app. That works until you need rotation, revocation, audit logs, or a security review.

A better pattern is to treat the password as a just-in-time input and the session as the reusable artifact.

The pattern: read the secret once, keep the session

For browser automation, you usually do not need the password on every run. You need it when no valid session exists.

The flow looks like this:

Store the password in a vault your team already controls, such as 1Password, Bitwarden, Vault, or AWS Secrets Manager.
Give the automation service a scoped token that can read only the required item.
When the session is missing or expired, read the credential into memory.
Log in with Playwright, Puppeteer, Selenium, or your browser driver of choice.
Persist the resulting session state.
Drop the password from memory and never write it to your database or logs.

Wire applies this model to authenticated browser automations by reading from 1Password when it needs to refresh a login, then running future tasks from the saved session rather than from the password.

The important distinction is this: your application stores something revocable and scoped, not the user's actual credential.

What this looks like in code

Here is a simplified Playwright example. It uses the 1Password CLI to fetch a login item, but the shape is the same if you use a vault SDK or a cloud secrets manager.

import { chromium } from 'playwright';
import { execFile } from 'node:child_process';
import { promisify } from 'node:util';
import fs from 'node:fs/promises';

const execFileAsync = promisify(execFile);

async function readLoginFrom1Password(itemId: string) {
  const { stdout } = await execFileAsync('op', [
    'item',
    'get',
    itemId,
    '--format',
    'json'
  ], {
    env: {
      ...process.env,
      OP_SERVICE_ACCOUNT_TOKEN: process.env.OP_SERVICE_ACCOUNT_TOKEN
    }
  });

  const item = JSON.parse(stdout);

  const username = item.fields.find((f: any) => f.id === 'username')?.value;
  const password = item.fields.find((f: any) => f.id === 'password')?.value;

  if (!username || !password) {
    throw new Error('Vault item is missing username or password');
  }

  return { username, password };
}

async function loginAndSaveSession(itemId: string, sessionPath: string) {
  const { username, password } = await readLoginFrom1Password(itemId);

  const browser = await chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();

  await page.goto('https://example.com/login');
  await page.fill('input[name="email"]', username);
  await page.fill('input[name="password"]', password);
  await page.click('button[type="submit"]');

  await page.waitForURL('https://example.com/dashboard', { timeout: 30_000 });

  await context.storageState({ path: sessionPath });
  await browser.close();
}

async function runAuthenticatedTask(sessionPath: string) {
  const browser = await chromium.launch();
  const context = await browser.newContext({ storageState: sessionPath });
  const page = await context.newPage();

  await page.goto('https://example.com/dashboard');

  if (page.url().includes('/login')) {
    throw new Error('Session expired or rejected');
  }

  // Continue with the authenticated workflow.

  await browser.close();
}

In production, I would avoid shelling out if a maintained SDK exists, but the example makes the boundary clear. The password enters memory for one login operation. The durable artifact is the browser storage state, not the password.

For Playwright, storageState usually contains cookies and localStorage. Treat it like a credential. Encrypt it at rest, restrict access to it, and expire it when the user revokes access.

The failure modes matter

This pattern does not remove authentication problems. It moves them to places you can reason about.

If the vault token is revoked, the next login refresh should fail clearly:

Error: failed to read credential from vault: access denied

Existing sessions may continue until the target site expires them. That is normal browser behavior. If you need immediate cutoff, delete the saved session state when vault access is revoked.

If the password rotates, the next login refresh should pick up the new value automatically. If you copied the password into your own database, rotation becomes a coordination problem. If you read from the vault at login time, rotation stays where it belongs.

If the target site changes its login page, your automation still breaks. You will see symptoms like:

TimeoutError: page.waitForURL: Timeout 30000ms exceeded

or:

Error: Session expired or rejected

That failure has nothing to do with secret storage. You still need monitoring around login success rate, selector failures, MFA challenges, and redirects to unexpected pages.

Audit logs should live outside your app too

One underrated benefit of vault-backed auth is that the vault can log every read. That gives your security team an audit trail in a system they already trust.

Your app should still log its own authentication events:

{
  "event": "credential_read_for_login",
  "vault": "1password",
  "item_id": "supplier-portal-prod",
  "workflow_id": "daily-inventory-sync",
  "timestamp": "2026-06-08T10:15:30Z"
}

Do not log usernames, passwords, session cookies, headers, or form payloads. Item IDs and workflow IDs are usually enough to debug access without leaking the secret itself.

Wire also relies on the vault provider's own activity log for credential reads, which means the audit trail does not only exist inside the automation platform.

What to store and what not to store

A reasonable implementation stores:

Encrypted session cookies or browser storage state
A pointer to the vault item
A scoped vault access token, encrypted separately
Metadata such as status, timestamps, and last refresh result

It should not store:

The password
A copy of the vault item contents
Screenshots of login pages containing filled password fields
Request logs containing auth headers or cookies
Full browser traces from login unless you scrub them carefully

The awkward part is the scoped vault token. It is still sensitive. If someone steals it, they may be able to read the same vault items your automation can read. Limit its scope, store it with envelope encryption, rotate it, and make revocation part of your offboarding process.

A practical way to start

If you already run authenticated automation, inspect where credentials live today. Search your database schema, environment variables, logs, and browser trace storage. Then change one workflow so it reads the password from a vault only when the saved session has expired. Keep the session encrypted, add a clear Session expired or rejected path, and verify that rotating the password in the vault does not require a deploy.

Why Cloudflare Breaks Proxy-Only Scrapers

Anakin — Wed, 10 Jun 2026 10:00:01 +0000

You rotate residential proxies, set a Chrome user agent, and still get a 403. Or worse, you get a 200 response that contains a Cloudflare challenge page instead of the product, article, or search result you expected. That usually means the site is not blocking just your IP address.

The proxy is only one part of the fingerprint

A proxy changes where the request comes from. It does not make your HTTP client behave like Chrome.

This is the common version of the problem:

import requests

url = "https://example.com/product/123"

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/121.0.0.0 Safari/537.36"
    )
}

proxies = {
    "https": "http://user:pass@residential-proxy.example:8000"
}

r = requests.get(url, headers=headers, proxies=proxies, timeout=30)

print(r.status_code)
print(r.text[:500])

The request might return:

403
<!DOCTYPE html><html><head><title>Just a moment...</title>

Or it may return 200, but the body still contains cf-chl, turnstile, or Just a moment. If your scraper only checks the status code, it will treat a block page as a successful scrape.

The user agent header says Chrome, but the rest of the connection does not. Cloudflare can inspect details such as:

TLS Client Hello ordering
Cipher suites and extensions
HTTP/2 SETTINGS frames
Header ordering and casing
JavaScript API behavior
Canvas, WebGL, and font fingerprints
Cookie and session consistency across requests

Python requests, Go net/http, and many scraping libraries do not emit the same low-level network fingerprint as a real browser. A residential IP does not fix that mismatch.

Cloudflare has different layers

It helps to separate the protection types, because they fail in different ways.

Basic challenge pages

These are the interstitial pages that run JavaScript before allowing the request through. A plain HTTP client usually gets stuck here because it does not execute the challenge.

Symptom: HTML contains Just a moment, cf-browser-verification, or challenge scripts instead of the target content.

Turnstile

Turnstile is Cloudflare's challenge widget. Sometimes it runs invisibly. Sometimes it requires interaction.

Symptom: the page renders, but the content remains gated until a token appears or a form submission completes.

Bot Management

Bot Management is the harder case. It can block before your scraper reaches useful HTML because the browser, TLS, protocol, and behavior signals do not line up.

Symptom: rotating IPs does not improve success rate much, and failures happen early with 403s, redirects, or repeated challenge loops.

This is where a browser-rendering approach often becomes necessary, not because browsers are magic, but because the target expects a real browser-shaped session.

Test for content, not just status codes

A basic verification loop should check three things:

Did the request return a 200?
Does the body contain the content you need?
Does the body contain signs of a block page?

For example:

from bs4 import BeautifulSoup

BLOCK_MARKERS = [
    "Just a moment",
    "cf-chl",
    "cf-browser-verification",
    "turnstile",
    "Attention Required!",
]

def classify_response(html: str, expected_selector: str) -> str:
    if any(marker in html for marker in BLOCK_MARKERS):
        return "blocked"

    soup = BeautifulSoup(html, "html.parser")
    if soup.select_one(expected_selector):
        return "ok"

    return "unknown"

result = classify_response(r.text, ".product-title")
print(result)

The unknown case matters. It catches layout changes, partial responses, empty states, and soft blocks that do not use obvious Cloudflare text.

Run this against 3 to 5 representative URLs before scaling. Include the pages that usually fail: product detail pages, search results, login-gated pages, and paginated flows. Ten successful requests against a homepage do not tell you much.

When proxies are still enough

Proxy rotation still works when the site mostly enforces IP reputation or rate limits. If you scrape static pages, public listings, or lightly protected endpoints, a browser may be unnecessary overhead.

The tradeoff is simple:

Proxy-only scraping is faster and cheaper
Browser rendering handles more protection layers
Browser sessions cost more and add latency
CAPTCHA and login flows need explicit handling
Multi-page workflows need cookie and storage persistence

The mistake is treating proxies and browsers as interchangeable. They solve different parts of the fingerprint.

When you have targets with mixed protection levels, Wire provides browser rendering, proxy-based scraping, and structured extraction behind one scraping API, so the mode can match the actual failure pattern instead of forcing every page through the same path: Wire.

A practical rule of thumb

Start with the cheapest thing that can prove it returns real content.

Use HTTP plus proxies if the target returns stable HTML and your validation passes. Move to browser rendering when you see challenge pages, Turnstile, session-sensitive navigation, or failures that do not improve with IP rotation. Keep the validation code in production either way, because a 200 response with a challenge page is still a failed scrape.

The full breakdown is here if you want the complete picture.

How Scraping API Pricing Changes Once You Need Browser Sessions

Anakin — Mon, 08 Jun 2026 10:00:01 +0000

A scraping job that looks cheap in development can get expensive in production. The usual surprise is not the price of one request. It is the number of billable attempts required to finish one logical job, especially when Cloudflare or another bot protection layer forces you into browser rendering.

Per-request billing is easy to reason about

Per-request pricing charges for each URL fetch. If you scrape one page and get one response, the model is clean.

For example, assume a provider charges $0.065 per rendered request:

function perRequestCost({ pages, pricePerRequest }) {
  return pages * pricePerRequest;
}

console.log(perRequestCost({
  pages: 10_000,
  pricePerRequest: 0.065
}));

// 650

That is predictable. It also scales linearly with page count.

This works well for jobs like:

Fetch one article URL
Render one search result page
Take one screenshot
Extract one public profile

It works less well when one job contains many pages from the same site. If you need 50 pages behind the same Cloudflare session, billing each page independently can waste the session state you already paid to establish.

Browser sessions change the unit of work

A browser session has startup cost. It needs to launch, negotiate protection, store cookies, run JavaScript, and keep state. Once that session exists, the next page in the same workflow may be much cheaper than starting over.

Session-duration billing charges for time instead of page count. For example, imagine a browser provider bills one credit per two-minute interval, rounded up.

function sessionCost({ sessions, minutesPerSession, creditsPerInterval, intervalMinutes }) {
  const intervalsPerSession = Math.ceil(minutesPerSession / intervalMinutes);
  return sessions * intervalsPerSession * creditsPerInterval;
}

const cost = sessionCost({
  sessions: 200,
  minutesPerSession: 10,
  creditsPerInterval: 1,
  intervalMinutes: 2
});

console.log(cost);
// 1000 credits

If each 10-minute session scrapes 50 pages, those 200 sessions cover 10,000 pages. The important metric becomes pages per session, not just requests per page.

That model rewards batching. It punishes slow navigation, unnecessary waits, and workflows that open a new browser for every URL.

Wire uses browser-session billing for browser jobs and exposes the work as async jobs, which fits long-running Cloudflare workflows where you care about job completion, retries, and session state rather than a single HTTP response: Wire.

Failed attempts are part of the cost

Pricing pages often show the happy path. Production scrapers spend money on failures too.

You need to know whether the provider charges for:

403 responses
Cloudflare challenge pages
Browser timeouts
CAPTCHA failures
Retries
Empty responses
Pages where extraction fails after rendering succeeds

A failed scrape is not always a failed HTTP request. This response is technically successful from an HTTP perspective:

HTTP/2 200
content-type: text/html

<html>
  <title>Just a moment...</title>
  <script src="/cdn-cgi/challenge-platform/..."></script>
</html>

If your billing model charges per response, you may pay for that. If your extraction pipeline treats it as success, you may also store bad data.

Track cost per successful extraction, not cost per request.

function costPerSuccessfulExtraction({ totalCost, successfulExtractions }) {
  if (successfulExtractions === 0) return Infinity;
  return totalCost / successfulExtractions;
}

console.log(costPerSuccessfulExtraction({
  totalCost: 650,
  successfulExtractions: 8200
}));

// 0.07926829268292683

That number is more useful than advertised request price because it includes blocks, retries, and parsing failures.

What to measure before choosing a model

Run the same small benchmark against each provider you are considering. Use real target URLs, not simple test pages.

For each run, record:

Number of attempted pages
Number of pages with expected content
Number of challenge or block pages
Median and p95 latency
Total billed units
Cost per successful extraction
Whether cookies and local storage persisted across pages

A simple CSV is enough:

provider,mode,pages,successes,blocks,p95_ms,billed_units,total_cost
provider_a,per_request,100,83,17,9400,100,6.50
provider_b,browser_session,100,96,4,18200,12,unknown

The p95 matters because browser rendering can be much slower than proxy-only scraping. If your job runs inside a queue with timeouts, a technically successful browser scrape may still fail your workflow.

Choosing the cheaper option

Use per-request billing when each page stands alone and protection is light. It is simple, and simple is good when the workload fits.

Use browser-session billing when you need to keep state across many pages: authenticated dashboards, carts, search pagination, product catalogs, or sites that challenge new sessions aggressively.

Use proxy bandwidth or enterprise proxy pricing when your team already manages crawling infrastructure and mainly needs IP diversity. The tradeoff is that bandwidth pricing can be harder to predict because page size, compression, retries, and geography all affect the bill.

Before you commit, run a 100-page test and calculate cost per successful extraction. That one number will tell you more than the pricing table.

The full breakdown is here if you want the complete picture.

When an Actor Platform Is Too Much for an LLM Scraping Task

Anakin — Fri, 05 Jun 2026 10:00:01 +0000

You start with a simple feature: give an LLM a URL, extract the useful data, and pass structured fields into the next prompt or tool call. Then the scraping layer grows its own lifecycle. You have runs, datasets, queues, retries, webhooks, SDK objects, and output formats that differ by target site. None of that is wrong, but it may be the wrong shape for the job.

The real decision is lifecycle shape

There are two common models for web extraction tools.

The first is the orchestration platform model. You configure a scraper, run it as a job, store output in a dataset, and maybe chain it with another job. Apify is a good example of this model with Actors, scheduled runs, dataset storage, proxies, and a large marketplace.

The second is the direct extraction API model. You send a request, get a job ID if it is async, poll or wait for the result, and receive JSON you can pass downstream.

For LLM workflows, this distinction matters more than the brand of the tool. An agent usually wants a narrow contract:

{
  "url": "https://example.com/product/123",
  "fields": ["title", "price", "availability", "description"]
}

And it wants a predictable response:

{
  "title": "Mechanical Keyboard",
  "price": "$129.00",
  "availability": "in_stock",
  "description": "Hot-swappable keyboard with...",
  "source_url": "https://example.com/product/123"
}

If your integration has to understand scraper-specific datasets, pagination, run logs, and output schemas before it can hand data to the model, you have added a pipeline where you may only need a tool call.

Wire is one example of the direct extraction API shape for LLM workflows: submit a task over REST, poll by job ID, and receive structured JSON back.

Where orchestration helps

Orchestration platforms make sense when the scraping system is a product in its own right.

Use that model when you need:

scheduled nightly or weekly crawls
reusable scrapers shared across teams
dataset storage and historical comparisons
proxy configuration across many geographies
chained jobs such as scrape, enrich, dedupe, export
operational dashboards for non-agent workflows

In that world, a queue is not overhead. It is the system. You want retries, logs, concurrency controls, and storage because your application depends on repeatable batch execution.

The tradeoff is integration surface area. A small LLM feature can end up with code like this:

type ScrapeRun = {
  runId: string;
  datasetId: string;
};

type DatasetItem = Record<string, unknown>;

async function getFirstDatasetItem(run: ScrapeRun): Promise<DatasetItem> {
  const dataset = await fetchDataset(run.datasetId);
  const items = await dataset.listItems();

  if (!items.length) {
    throw new Error(`Scrape finished but dataset ${run.datasetId} was empty`);
  }

  return items[0];
}

The failure mode is also awkward. The job can succeed while the dataset contains no useful item. Your application sees undefined, or worse, passes an empty object into the LLM and gets a confident answer based on missing data.

That is not a tool failure in the strict sense. It is a contract failure.

A better contract for agent tools

For an LLM tool, design around the data contract first and the scraping provider second.

A simple interface might look like this:

type ExtractionRequest = {
  targetUrl: string;
  schema: Record<string, "string" | "number" | "boolean" | "string[]">;
};

type ExtractionResult = {
  status: "completed" | "failed";
  data?: Record<string, unknown>;
  error?: {
    code: string;
    message: string;
  };
  metadata: {
    sourceUrl: string;
    executionMs?: number;
  };
};

Then hide the provider behind an adapter:

async function extractForAgent(req: ExtractionRequest): Promise<ExtractionResult> {
  const response = await fetch(process.env.EXTRACTION_API_URL!, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "X-API-Key": process.env.EXTRACTION_API_KEY!
    },
    body: JSON.stringify(req)
  });

  if (response.status === 401) {
    return {
      status: "failed",
      error: { code: "AUTH_REQUIRED", message: "Invalid or missing API key" },
      metadata: { sourceUrl: req.targetUrl }
    };
  }

  if (!response.ok) {
    return {
      status: "failed",
      error: { code: "EXTRACTION_FAILED", message: await response.text() },
      metadata: { sourceUrl: req.targetUrl }
    };
  }

  const data = await response.json();

  return {
    status: "completed",
    data,
    metadata: { sourceUrl: req.targetUrl }
  };
}

The important part is not the exact endpoint. The important part is that the agent sees one stable contract. It does not know about Actors, datasets, run IDs, browser sessions, or selector logic.

If the provider requires async polling, keep that inside the adapter too:

async function pollJob(jobUrl: string, apiKey: string) {
  for (let attempt = 0; attempt < 20; attempt++) {
    const res = await fetch(jobUrl, { headers: { "X-API-Key": apiKey } });
    const body = await res.json();

    if (body.status === "completed") return body.data;
    if (body.status === "failed") throw new Error(body.error?.message ?? "Job failed");

    await new Promise(resolve => setTimeout(resolve, 1000));
  }

  throw new Error("Extraction timed out after 20 seconds");
}

This prevents a common bug in agent systems: letting provider-specific states leak into prompt logic. Your LLM should not need to reason about RUNNING, READY, TIMED-OUT, and dataset item count = 0 unless those states mean something to the user.

Structured output beats raw HTML for most LLM paths

Raw HTML is useful if you are building a crawler or debugging selectors. It is less useful when the next step is a function call, embedding job, or RAG chunk.

A selector-based scraper can fail quietly when a site changes markup:

const price = document.querySelector(".product-price")?.textContent;

// Later:
if (!price) {
  // Was the product free, unavailable, blocked, or did the selector break?
}

That ambiguity matters. If the extraction layer cannot distinguish “field not present” from “page blocked” from “selector stale”, your LLM workflow has to guess.

A better failure response looks like this:

{
  "status": "failed",
  "error": {
    "code": "EXTRACTION_FAILED",
    "message": "Could not locate price field on rendered page"
  }
}

That gives your application something deterministic to do: retry, ask for a different URL, fall back to another provider, or tell the user what happened.

Wire exposes structured error states for authentication, credit, and execution failures, which fits this adapter pattern better than parsing multi-stage job logs in the agent layer.

How to choose without overfitting to one tool

Pick an orchestration platform if your main problem is operating scraping workflows at scale. Pick a direct extraction API if your main problem is feeding structured web data into an LLM with as little glue code as possible.

The quick test is this: if you removed the LLM from your application, would you still need queues, scheduled runs, stored datasets, and shared scraper assets? If yes, orchestration probably belongs in your architecture. If no, keep the extraction layer small and make the contract boring.

A practical next step: write the ExtractionRequest and ExtractionResult types for your app before choosing a provider, then build one adapter around them and make sure failed extraction never reaches your prompt as empty data.

Choosing the Right Scraping Interface for LLM Workflows

Anakin — Thu, 04 Jun 2026 10:00:01 +0000

You build an LLM feature that needs data from a web page, and the scraping part quietly becomes half the project. The model does not want HTML. It wants clean fields: title, price, author, company name, profile URL, publication date. But many scraping tools give you a job queue, a browser runtime, a dataset store, logs, retries, and raw output you still need to normalize.

That extra machinery is not always bad. It depends on what problem you actually have.

Two different jobs get called “scraping”

A lot of confusion comes from treating these as the same task:

Running a repeatable data pipeline across thousands of pages
Fetching structured context for an LLM or agent step

The first one benefits from orchestration. You may need scheduled runs, proxy pools, task queues, retries, storage, webhooks, and audit logs. Apify is a good example of this model: Actors encapsulate site-specific scraping logic, and the platform handles a lot of the operational surface area around running them.

The second one usually needs a much narrower interface:

input:  URL or search/query parameters
output: typed JSON that can be passed to the next tool call

If your agent has to scrape a company profile before drafting a summary, you probably do not want to browse a marketplace, configure an Actor input schema, wait for a dataset export, and then write another parser for the result. You want a small contract with predictable failure modes.

Wire by Anakin exposes this direct submit-and-poll pattern for web extraction tasks, returning structured JSON through a REST API instead of making the caller manage an actor lifecycle.

The integration shape matters more than the vendor name

For LLM workflows, I usually look at four things before choosing a scraping service.

What does the API return?

Raw HTML is flexible, but it pushes parsing onto your app. Markdown is better for summarization and RAG ingestion, but it may still leave you extracting fields with prompts or regex.

Structured JSON is the easiest output to compose with tool calls:

{
  "title": "Example article",
  "author": "Jane Doe",
  "published_at": "2026-01-12",
  "summary": "...",
  "links": [
    { "label": "Docs", "url": "https://example.com/docs" }
  ]
}

The catch is that structured output only helps if the schema stays stable. If one target returns published_at, another returns date, and a third nests it under metadata.created, your agent code still needs normalization.

How many states do you have to handle?

A simple async extraction flow usually has three states:

processing -> completed
processing -> failed
processing -> timed_out

That is easy to wrap. A platform workflow can have more places to fail: Actor startup, browser launch, proxy allocation, dataset write, webhook delivery, result export, downstream transform. Those states are useful when you need observability, but they are noisy when you only need one result.

A minimal polling wrapper looks like this:

import time
import requests

API_KEY = "..."
BASE_URL = "https://api.example.com"


def submit_extraction(url: str) -> str:
    response = requests.post(
        f"{BASE_URL}/v1/tasks",
        headers={
            "X-API-Key": API_KEY,
            "Content-Type": "application/json",
        },
        json={
            "action": "extract_article",
            "params": {"url": url},
        },
        timeout=10,
    )
    response.raise_for_status()
    return response.json()["job_id"]


def wait_for_result(job_id: str, timeout_seconds: int = 60):
    deadline = time.time() + timeout_seconds

    while time.time() < deadline:
        response = requests.get(
            f"{BASE_URL}/v1/jobs/{job_id}",
            headers={"X-API-Key": API_KEY},
            timeout=10,
        )
        response.raise_for_status()
        payload = response.json()

        if payload["status"] == "completed":
            return payload["data"]

        if payload["status"] == "failed":
            code = payload.get("error", {}).get("code", "UNKNOWN")
            message = payload.get("error", {}).get("message", "No details")
            raise RuntimeError(f"Extraction failed: {code}: {message}")

        time.sleep(2)

    raise TimeoutError(f"Extraction job {job_id} did not finish in time")


data = wait_for_result(submit_extraction("https://example.com/article"))
print(data)

This is not complicated, and that is the point. The less custom lifecycle code you write, the easier it is to swap providers later.

How does it fail?

Selector-based scrapers often fail in quiet ways. A CSS selector stops matching, the scraper returns an empty string, and your LLM confidently reasons over missing data. JavaScript-heavy pages add another failure mode: the HTML response exists, but the content you need only appears after client-side rendering.

You want failures that your application can detect:

{
  "status": "failed",
  "error": {
    "code": "EXTRACTION_EMPTY",
    "message": "The page loaded, but no article body was found"
  }
}

That is much easier to handle than a successful 200 OK containing a blank field. Your agent can retry, ask for a different URL, fall back to search, or skip that source.

Wire uses API-key authenticated REST calls and structured job errors, which fits this kind of provider wrapper without requiring a language-specific SDK.

When orchestration is still the right choice

Do not replace an orchestration platform just because the API feels heavier. It earns its keep when you need scheduled jobs, large shared scraper catalogs, proxy rotation, persisted datasets, team-level reuse, and long-running batch workflows.

If you run nightly competitor monitoring across 50,000 product pages, a queue-backed platform with storage and retries is probably the right shape. If you need to archive every raw response for compliance or debugging, dataset storage matters. If multiple teams depend on the same site-specific scraper, packaging that logic as a reusable unit can save work.

The tradeoff is complexity. Your application now understands platform concepts: tasks, runs, datasets, webhooks, actor versions, resource limits. That is acceptable for data infrastructure. It can be too much for an LLM feature that needs one clean object.

A practical rule of thumb

Use orchestration when scraping is the product or a core data pipeline.

Use a direct extraction API when scraping is just one tool call inside a larger LLM workflow.

Before committing, write the smallest adapter you can: submit a URL, poll for completion, validate the JSON schema, and force at least three failure cases in tests: timeout, empty extraction, and provider error. If that adapter stays small, the integration shape is probably right.

When scraping orchestration is the wrong abstraction for LLM workflows

Anakin — Wed, 03 Jun 2026 10:00:01 +0000

A lot of LLM workflows start with the same small problem: the model needs fresh data from a web page. Then the integration grows sideways. You add a scraper, a queue, a dataset store, polling logic, retries, and a parser. By the end, the code that moves data around is larger than the code that uses the data.

This is not because scraping platforms are bad. It is because they solve a broader problem than many LLM apps actually have.

The abstraction mismatch

Platforms like Apify are built around actors: reusable scraping or automation jobs with inputs, runs, logs, datasets, scheduling, and platform-managed execution. That model makes sense when you run recurring jobs across many targets, chain multiple scraping tasks, or need shared actors across a team.

For example, a batch pipeline might look like this:

schedule -> run actor -> wait for completion -> read dataset -> normalize rows -> store results -> trigger downstream job

That is useful if you are refreshing competitor pricing every night or maintaining a long-lived dataset.

An LLM tool call usually looks different:

prompt -> fetch one page -> extract fields -> pass JSON back to the model

If you use a full actor lifecycle for that second case, you pay for concepts you may not need: actor discovery, input schemas, run state, dataset retrieval, and actor-specific output formats. The failure modes also spread out. A run can succeed while the dataset is empty. A page can render differently and produce partial data. A parser can return HTML where your downstream tool expects JSON.

That is where the abstraction matters more than the vendor.

Design the tool around the shape your agent needs

For most agentic workflows, the cleanest internal interface is not “run scraper X.” It is “given this target and extraction intent, return typed data or a typed error.”

Something like this:

type ExtractRequest = {
  url: string;
  schema: Record<string, string>;
};

type ExtractResult =
  | {
      ok: true;
      data: Record<string, unknown>;
      sourceUrl: string;
    }
  | {
      ok: false;
      error: "AUTH" | "TIMEOUT" | "BLOCKED" | "EMPTY_RESULT" | "INVALID_OUTPUT";
      message: string;
    };

Then hide the provider behind an adapter:

async function extractPage(req: ExtractRequest): Promise<ExtractResult> {
  const res = await fetch(process.env.EXTRACT_API_URL!, {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.EXTRACT_API_KEY}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify(req)
  });

  if (res.status === 401) {
    return { ok: false, error: "AUTH", message: "Invalid API key" };
  }

  if (res.status === 408 || res.status === 504) {
    return { ok: false, error: "TIMEOUT", message: "Extraction timed out" };
  }

  if (!res.ok) {
    return { ok: false, error: "BLOCKED", message: await res.text() };
  }

  const body = await res.json();

  if (!body || Object.keys(body.data ?? {}).length === 0) {
    return { ok: false, error: "EMPTY_RESULT", message: "No structured fields returned" };
  }

  return {
    ok: true,
    data: body.data,
    sourceUrl: req.url
  };
}

The important part is not the exact provider. The important part is that your LLM application receives a predictable result. The model should not need to know whether the data came from a browser automation run, a marketplace actor, a custom scraper, or a REST extraction endpoint.

Wire by Anakin uses this direct extraction shape: submit a task over REST, poll the job, and get structured JSON back for the next tool call.

Polling is fine if you keep it contained

A lot of scraping APIs are async because pages take time to load, render, and extract. That is fine. The mistake is letting async job management leak through your whole application.

Keep polling inside the adapter:

async function pollJob(jobUrl: string, apiKey: string, timeoutMs = 30000) {
  const started = Date.now();

  while (Date.now() - started < timeoutMs) {
    const res = await fetch(jobUrl, {
      headers: { "X-API-Key": apiKey }
    });

    const body = await res.json();

    if (body.status === "completed") return body.data;

    if (body.status === "failed") {
      throw new Error(body.error_code ?? "EXTRACTION_FAILED");
    }

    await new Promise(resolve => setTimeout(resolve, 1000));
  }

  throw new Error("EXTRACTION_TIMEOUT");
}

Your agent code should call extractPage() and receive data or an error. It should not manage run IDs, dataset IDs, actor logs, and retry policy unless those concepts matter to the product.

When orchestration is the right choice

A direct extraction API is not always better. If you need scheduled scraping, dataset versioning, large proxy pools, or reusable workflows that non-LLM systems consume, an actor-based platform can be the better fit.

Apify’s model works well when you want a managed scraping pipeline rather than a single extraction call. The actor marketplace also helps when a specific site already has a maintained actor and your team is comfortable with that platform’s run and dataset model.

The tradeoff is coupling. Your application starts to understand provider-specific concepts: actors, runs, datasets, schemas, and logs. That may be acceptable for a data engineering pipeline. It is often unnecessary for a chat agent or RAG ingestion path that just needs structured fields.

Wire is aimed at the lower-overhead case, where the integration contract is HTTP in, structured JSON out, without adding a provider SDK to every runtime.

A practical way to decide

Before choosing a scraping tool, write the interface your application wants. Not the provider API. Your API.

Ask these questions:

Does the app need one result now, or a scheduled dataset later?
Will the LLM consume structured fields directly?
Do you need raw HTML for audit/debugging?
Can an empty result be retried, or does it need human review?
Are provider-specific concepts allowed to leak into business logic?

If the workflow is prompt-driven and short-lived, keep the extraction layer small and typed. If the workflow is recurring, shared, and operationally complex, orchestration may be worth the extra surface area.

A good next step is to implement a provider-neutral extractPage() interface, run it against three real URLs your app depends on, and log every failure as one of your own error types. That will tell you quickly whether you need an extraction API or a full scraping platform.

AI agents don't always need browsers to read the web

Anakin — Tue, 02 Jun 2026 09:30:00 +0000

You build an agent that needs data from a website. The first version uses Playwright because it works locally: open the page, wait for the selector, read the DOM. Then you run it in production and the failures change. Jobs hang because a page never reaches networkidle. Containers run out of memory when several browsers start at once. A single slow site blocks the whole agent loop.

The problem usually is not the model. It is the shape of the web access layer.

Browser automation is a high-fidelity tool, not a default

Playwright and Puppeteer are good when you need a browser. They execute client-side JavaScript, keep cookies, click buttons, fill forms, handle navigation, and give you screenshots when something goes wrong.

That fidelity costs you CPU, memory, and time.

A common failure looks like this:

TimeoutError: page.waitForSelector: Timeout 30000ms exceeded.

Or, under load:

Error: browserType.launch: Target page, context or browser has been closed

Those are not unusual edge cases. They happen because a browser session is a stateful resource. If an agent needs to fetch 200 pages, you either run those pages serially and wait a long time, or you run browsers in parallel and pay for the resources. You also need queues, retries, proxy handling, session cleanup, and observability around every job.

Use browser automation when the task really needs browser behavior: interactive forms, visual validation, authenticated flows that depend on client-side state, or data that only appears after complex JavaScript execution.

If the agent just needs structured text, metadata, prices, reviews, or search results, a browser is often too much machinery.

Prefer an async extraction boundary

A better pattern for many agent workflows is: submit an extraction job, get a job ID, keep doing other work, then poll for a typed result.

That boundary matters because web requests have unpredictable duration. Your agent should not block its reasoning loop on a single HTTP request that might take 45 seconds or fail because of a rate limit.

The generic flow looks like this:

POST /tasks
-> 202 Accepted { "job_id": "abc", "status": "processing" }

GET /jobs/abc
-> { "status": "completed", "data": { ... } }

Wire uses this kind of async job model for web extraction actions, returning a job_id first and structured results later when the extraction completes.

Here is a small poller that works for this style of API. The important parts are terminal states, Retry-After, and a maximum wait so your agent does not loop forever.

import time
import httpx

class JobFailed(Exception):
    pass

def poll_job(job_url: str, headers: dict, max_attempts: int = 12) -> dict:
    for attempt in range(max_attempts):
        response = httpx.get(job_url, headers=headers, timeout=10)

        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After")
            delay = int(retry_after) if retry_after else min(2 ** attempt, 30)
            time.sleep(delay)
            continue

        response.raise_for_status()
        payload = response.json()

        status = payload.get("status")

        if status == "completed":
            return payload["data"]

        if status == "failed":
            raise JobFailed(payload.get("error", "job failed"))

        if status != "processing":
            raise RuntimeError(f"unknown job status: {status}")

        time.sleep(min(2 ** attempt, 30))

    raise TimeoutError("job did not complete before max_attempts")

This pattern also makes retries safer. Retrying a synchronous scrape can accidentally duplicate side effects if the target workflow submits a form or changes state. Polling a job ID is usually idempotent. You can call GET /jobs/{id} as many times as needed without starting the work again.

Return data the agent can use directly

Agents are bad places to hide parsing logic.

If your tool returns raw HTML, the model has to infer structure every time. That increases token usage and makes failures harder to diagnose. A better extraction layer returns typed fields:

{
  "title": "Example thread title",
  "author": "user123",
  "score": 2847,
  "comments": [
    {
      "author": "another_user",
      "body": "I hit the same issue with polling.",
      "created_at": "2026-01-12T10:15:00Z"
    }
  ]
}

This gives your agent a stable contract. The LLM can reason over score, comments, and created_at instead of guessing where those values live in the DOM.

The tradeoff is that typed extraction depends on schemas. When a site changes markup or behavior, someone still has to update the extractor. The benefit is that the fix happens in one place instead of inside every agent prompt or parser.

Wire exposes a catalog of site-specific actions with declared auth requirements and credit costs, which is one way to move platform-specific parsing and session handling out of the agent code.

Even if you build this yourself, the same design applies: define actions with input parameters, output schemas, retry behavior, and authentication mode. Treat web access like an API contract, not like an improvised browser script.

Be explicit about when to fall back to a browser

API-first extraction is not a replacement for all browser automation.

Use an extraction API when:

The data is mostly text or structured records
You need many pages per workflow
Latency and resource usage matter
The agent can continue working while jobs run
The output schema is predictable

Use a browser when:

The workflow requires clicks, forms, or navigation state
You need screenshots or visual assertions
The page depends on complex client-side rendering
CAPTCHA or interactive auth is part of the flow
You are testing the user experience itself

Mixing both is fine, but do it deliberately. For example, use API extraction for 95% of product pages and reserve Playwright for the few sites where data only appears after an interactive flow. What you want to avoid is sending every request through a browser because the first prototype did.

The architecture is the useful part

For production agents, the useful pattern is not “scrape without a browser” as a slogan. It is a concrete boundary:

Submit work asynchronously
Poll with backoff
Respect Retry-After
Return typed JSON
Track terminal failure states
Keep browser automation for tasks that need browser fidelity

A practical next step: take one existing Playwright scraper, measure how often it only reads static or semi-structured data, and replace that path with an async job interface that returns a typed JSON schema. Keep the browser path only for the cases that actually need it.

When AI Agents Should Stop Using Browsers for Web Data

Anakin — Mon, 01 Jun 2026 09:30:02 +0000

You start with Playwright because it works. The agent needs data from a page, so you give it a browser, navigate to the URL, wait for selectors, extract text, and move on. Then the workflow grows from 3 pages to 300. Suddenly you are debugging Navigation timeout of 30000 ms exceeded, Target closed, rate limits, memory pressure, and a queue full of half-dead browser sessions.

The problem is not that browser automation is bad. It is that a browser is often the wrong abstraction for getting structured data into an AI system.

Browsers are useful, but expensive

A headless browser gives you high fidelity. It runs JavaScript, stores cookies, clicks buttons, submits forms, follows client-side routing, and sees the page roughly as a user would.

That matters for:

multi-step login flows
pages that render all useful data client-side
visual testing
checkout or booking flows
workflows that need clicks, hovers, uploads, or form submission

But if your agent only needs titles, prices, comments, abstracts, availability, or review text, a full browser can become unnecessary overhead.

Each browser session consumes CPU and memory. Parallel extraction means parallel browser contexts or instances. You also have to manage lifecycle issues: browser startup, page crashes, timeouts, proxy assignment, retries, and cleanup.

A typical browser-based extraction loop looks like this:

import { chromium } from "playwright";

const browser = await chromium.launch();
const page = await browser.newPage();

try {
  await page.goto("https://example.com/product/123", {
    waitUntil: "networkidle",
    timeout: 30_000,
  });

  const title = await page.locator("h1").innerText();
  const price = await page.locator("[data-testid='price']").innerText();

  console.log({ title, price });
} finally {
  await browser.close();
}

This is fine for a small number of pages. At scale, the failure modes stack up. networkidle may never happen because analytics requests keep running. A selector change breaks extraction. A page crash loses the whole session. If the agent launches many of these calls at once, infrastructure becomes part of the reasoning loop whether you wanted it or not.

Structured extraction is a better default when the shape is known

For many agent workflows, the goal is not to interact with the site. The goal is to turn web content into typed data.

Instead of giving the agent a browser, give it an extraction API that returns JSON:

{
  "title": "Example product",
  "price": 42.99,
  "currency": "USD",
  "availability": "in_stock"
}

That changes the agent’s job. It no longer has to reason about selectors, loading states, cookie banners, or whether a button is visible. It receives data and decides what to do with it.

Wire is Anakin’s API layer for web actions, including catalog-based extractors that return structured data without making the agent manage browser sessions directly.

The broader pattern is what matters: move brittle web interaction out of the prompt and into deterministic infrastructure. Let code handle retries, parsing, authentication, and rate limits. Let the model handle ranking, summarizing, planning, or answering the user.

This works best when:

the data shape is predictable
you need many pages per workflow
visual fidelity does not matter
the agent can run jobs asynchronously
failed extractions can be retried or skipped

It works poorly when the site requires complex interaction, CAPTCHA solving, dynamic user-specific flows, or visual verification. In those cases, use a browser. Do not pretend JSON extraction replaces the entire browser automation stack.

Use async jobs instead of long HTTP requests

Long-running extraction should usually be asynchronous. A synchronous request ties your agent to an HTTP timeout. If the page is slow, the proxy retries, or the extraction spans multiple pages, you either block the agent or fail the request.

A better pattern is:

Submit a job.
Get a job_id.
Continue other work.
Poll until the job reaches completed or failed.
Feed the result back into the agent.

The API shape usually looks like this:

curl -X POST https://api.example.com/v1/tasks \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "extract.product",
    "params": {
      "url": "https://example.com/product/123"
    }
  }'

# 202 Accepted
# { "job_id": "job_abc123", "status": "processing" }

Then poll:

curl https://api.example.com/v1/jobs/job_abc123 \
  -H "Authorization: Bearer $API_KEY"

# 200 OK
# {
#   "status": "completed",
#   "data": {
#     "title": "Example product",
#     "price": 42.99
#   }
# }

Wire uses this same async job style for heavier web actions: submit work, receive a job id, and poll for structured results while the agent continues other tasks.

The important part is not the vendor-specific endpoint. It is the control flow. Agents often run several web lookups in parallel, and async jobs make that manageable.

Polling needs backoff and terminal states

Do not poll every 100ms. Also do not poll forever.

Your polling code should handle:

processing as non-terminal
completed as success
failed as terminal failure
429 with Retry-After
transient 5xx errors
a maximum attempt count or deadline

Example:

import time
import httpx

TERMINAL = {"completed", "failed"}


def poll_job(job_id: str, api_key: str, deadline_seconds: int = 120):
    url = f"https://api.example.com/v1/jobs/{job_id}"
    headers = {"Authorization": f"Bearer {api_key}"}
    started = time.monotonic()
    attempt = 0

    while time.monotonic() - started < deadline_seconds:
        resp = httpx.get(url, headers=headers, timeout=10)

        if resp.status_code == 429:
            retry_after = int(resp.headers.get("Retry-After", "1"))
            time.sleep(retry_after)
            continue

        if 500 <= resp.status_code < 600:
            time.sleep(min(2 ** attempt, 30))
            attempt += 1
            continue

        resp.raise_for_status()
        payload = resp.json()
        status = payload.get("status")

        if status == "completed":
            return payload["data"]

        if status == "failed":
            raise RuntimeError(payload.get("error", "extraction failed"))

        time.sleep(min(2 ** attempt, 30))
        attempt += 1

    raise TimeoutError(f"job {job_id} did not finish within {deadline_seconds}s")

This code is boring on purpose. The agent should not improvise retry policy in natural language. Put that behavior in normal application code where you can test it.

Pick the lowest-fidelity tool that works

A useful rule: use the lowest-fidelity web access method that gives correct data.

Start with direct APIs if the site provides them. Use structured extraction when the data is public or session-backed but predictable. Use browser automation when interaction or rendering fidelity is required.

Mixing all three is normal, but keep the boundaries clear. If every extraction path eventually falls back to a browser, you still need browser infrastructure. If most tasks return JSON and only a few need Playwright, your system gets simpler and cheaper to operate.

A practical next step: take one existing browser-based extraction in your agent stack and log what it actually uses from the page. If it only reads a handful of fields, replace that path with a JSON-producing function and keep the browser version as a fallback.