DEV Community: Anakin

What coding agents need from a real web data layer

Anakin — Thu, 09 Jul 2026 10:00:03 +0000

You ask a coding agent to compare three pricing pages, check a changelog, or pull current compliance dates. It comes back with something plausible, but one source was stale, another page was JavaScript-rendered, and the third was summarized so aggressively that the important caveat disappeared.

That is usually not a model problem. It is a data access problem.

Built-in web tools are lookup tools

Most agent harnesses have some form of web access now. Claude Code has WebSearch and WebFetch. Codex can run with cached, live, or disabled web search. Cursor and other IDE agents expose similar primitives.

These tools help, but they are not the same thing as a web data layer.

Search usually returns titles, URLs, and maybe short snippets. That is enough to decide what to fetch next, but not enough to reason over the source. Fetch tools often return a model-generated summary instead of the original page content. That keeps token usage down, but it also means the agent reasons over another model's interpretation.

The common failure modes are familiar:

Search result: "Pricing | Example SaaS"
Agent conclusion: "The Pro plan costs $49/month"
Actual page: price changed to $59/month behind client-side rendering
Fetched HTML: <div id="root"></div>

Or this:

HTTP 403
<title>Just a moment...</title>
Cloudflare challenge page returned instead of article content

If your agent cannot tell the difference between a real page and a challenge page, it will often continue anyway.

Treat web access as a small data pipeline

A useful web data layer has three separate jobs:

Search and return enough text to rank sources.
Fetch the actual page, including JavaScript-rendered content when needed.
Convert messy page content into typed data the agent can validate.

I like to make that boundary explicit:

type SearchResult = {
  url: string;
  title: string;
  snippet: string;
  date?: string;
};

type FetchResult = {
  url: string;
  status: "completed" | "failed" | "timeout";
  markdown?: string;
  html?: string;
  error?: string;
};

interface WebDataLayer {
  search(query: string, limit: number): Promise<SearchResult[]>;
  fetchPage(url: string, opts?: { renderJs?: boolean }): Promise<FetchResult>;
  extract<T>(input: string, schemaName: string): Promise<T>;
}

This shape matters because it gives the agent concrete states to handle. A failed fetch is not the same as an empty page. A rendered page is not the same as raw HTML. A snippet is not source content.

Anakin's remote MCP connector follows this split with search, scrape, crawl, and structured site actions, but the pattern is useful even if you build the tools yourself.

Async scraping is usually the right default

Fetching a static HTML page can be synchronous. Rendering a SPA, waiting for network calls, rotating proxies, or retrying after a bot challenge should not block an agent request forever.

A better scraper API returns a job ID, then lets the caller poll:

curl -X POST https://example.internal/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com/pricing","renderJs":true}'

# 202 Accepted
# {"jobId":"job_abc123","status":"pending"}

Then your agent-side wrapper handles completion, failure, and timeout explicitly:

async function waitForScrape(jobId: string, timeoutMs = 60_000) {
  const started = Date.now();

  while (Date.now() - started < timeoutMs) {
    const res = await fetch(`https://example.internal/scrape/${jobId}`);
    const job = await res.json();

    if (job.status === "completed") return job;

    if (job.status === "failed") {
      throw new Error(`Scrape failed: ${job.error ?? "unknown error"}`);
    }

    await new Promise(resolve => setTimeout(resolve, 2000));
  }

  throw new Error(`Scrape timed out after ${timeoutMs}ms: ${jobId}`);
}

This is less convenient than a single request, but it gives you better control. You can cap waiting time, log failure reasons, retry selectively, and avoid pretending that a challenge page is valid content.

The tradeoff is latency. Browser rendering can take seconds. For interactive coding sessions, you may want a fast path for normal pages and an async rendered path only when the raw fetch looks wrong.

A simple heuristic catches many misses:

function probablyNeedsBrowser(html: string) {
  return html.includes('<div id="root"></div>') ||
    html.includes('<div id="app"></div>') ||
    html.includes('Just a moment...') ||
    html.length < 1000;
}

It is not perfect. It is better than silently summarizing an empty shell.

Structured extraction should validate output

Agents are good at reading pages, but you should not let them invent data contracts on every run. If you need product names, prices, dates, or availability, define the shape and validate it.

import { z } from "zod";

const PricingPlan = z.object({
  name: z.string(),
  monthlyPriceUsd: z.number().nullable(),
  limits: z.array(z.string()),
  sourceUrl: z.string().url()
});

const PricingPlans = z.array(PricingPlan);

const raw = await web.extract<unknown>(markdown, "pricing_plans");
const plans = PricingPlans.parse(raw);

If parsing fails, you have a real error:

ZodError: Expected number, received string at [0].monthlyPriceUsd

That is much easier to debug than a paragraph saying "the pricing appears to be...".

For sites without APIs, selector-based scrapers often break when teams rename classes or move content into a new component. Wire addresses that specific case with pre-built read actions that return typed JSON for supported sites instead of asking the agent to infer structure from HTML.

Coverage is the obvious tradeoff. A catalog action is useful when it exists and stays maintained. A generic scraper still needs to handle everything else.

Expose the tools through MCP if your agent supports it

MCP is a practical way to give different agents the same tool boundary. The exact server can be your own service, a hosted connector, or a thin wrapper around internal APIs.

For example, a Claude Code setup might look like this:

claude mcp add --transport http webdata https://mcp.example.internal/mcp

Cursor uses JSON config instead:

{
  "mcpServers": {
    "webdata": {
      "url": "https://mcp.example.internal/mcp"
    }
  }
}

The important part is not the client. It is that the agent sees separate tools with clear behavior: search, fetchPage, crawl, and extractStructuredData. Do not hide every operation behind one vague "browse the web" function.

A reasonable starting point

Start small. Add search results with snippets, a page fetcher that can detect obvious bad HTML, and schema validation for one workflow you actually care about. Log the source URL, fetch mode, status, content length, and validation errors.

Then run your agent against five pages that have caused problems before: a static doc page, a SPA, a pricing page, a bot-protected page, and a page with stale indexed content. The gaps will show up quickly, and you can decide whether to add rendering, async jobs, crawling, or site-specific structured actions from there.

Polling Async Web Data Jobs Without Burning Your API Quota

Anakin — Thu, 02 Jul 2026 10:00:01 +0000

Your RAG pipeline looks fine in staging, then someone asks about a product price, policy update, or news item that changed three hours ago. The model answers from yesterday's vector index because the crawler runs nightly. Moving to live web retrieval fixes the freshness problem, but it introduces another one: slow, failure-prone network work now sits in the path of your AI system.

Treat web retrieval as a job, not a request

A synchronous scrape is tempting because it is easy to reason about:

request comes in -> fetch page -> parse page -> answer user

That falls apart when one target site takes 20 seconds, rate limits you, or returns a CAPTCHA page. Your worker sits there waiting, and your upstream timeout decides the result.

The more reliable pattern is:

submit job -> receive job_id -> poll status -> read result

This decouples the caller from the work. The orchestrator can submit several jobs, persist their IDs, and resume later if a process restarts. Most scraping and extraction APIs that do non-trivial work use some version of this shape.

For structured extraction tasks that return job IDs, Wire follows the same submit, poll, retrieve pattern, so the polling loop you write here applies beyond plain HTML scraping.

A typical submission response should look boring:

{
  "job_id": "job_123",
  "status": "pending"
}

The important part is that the API returns quickly. The expensive work happens elsewhere.

Poll slowly enough to matter

The easiest mistake is polling every 500ms because it feels responsive. If most jobs finish in 3 to 15 seconds, those early polls only return this:

{
  "job_id": "job_123",
  "status": "processing"
}

You gain nothing, but you spend rate-limit budget and sometimes billable API calls.

A practical default is:

Wait 3 to 5 seconds before the first poll for normal HTTP scraping.
Wait around 10 seconds before the first poll for browser-rendered pages.
Use exponential backoff after that.
Cap the interval so a long job does not disappear for too long.

Here is a TypeScript implementation I would be comfortable putting behind a queue worker:

class JobTimeoutError extends Error {}
class JobFailedError extends Error {}

type JobStatus<T> =
  | { status: 'pending' | 'processing' }
  | { status: 'completed'; result: T }
  | { status: 'failed'; error?: string };

async function sleep(ms: number) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

async function pollJob<T>(
  jobId: string,
  getStatus: (jobId: string) => Promise<JobStatus<T>>,
  options = {
    initialDelayMs: 5000,
    maxDelayMs: 60000,
    maxAttempts: 10
  }
): Promise<T> {
  let delay = options.initialDelayMs;

  for (let attempt = 0; attempt < options.maxAttempts; attempt++) {
    await sleep(delay);

    const job = await getStatus(jobId);

    if (job.status === 'completed') {
      return job.result;
    }

    if (job.status === 'failed') {
      throw new JobFailedError(job.error ?? `Job ${jobId} failed`);
    }

    delay = Math.min(delay * 2, options.maxDelayMs);
  }

  throw new JobTimeoutError(`Job ${jobId} did not finish after ${options.maxAttempts} polls`);
}

With a 5 second initial delay and a 60 second cap, the loop polls at roughly 5s, 10s, 20s, 40s, then 60s intervals. That is usually enough for web extraction jobs without hammering the status endpoint.

Separate retryable failures from permanent ones

Retries are not a moral good. They help when the failure is transient. They waste money and hide bugs when the request is wrong.

I usually classify failures like this:

Symptom	Likely cause	Action
`502 Bad Gateway`, `503 Service Unavailable`	Upstream or provider issue	Retry with backoff
Network timeout	Temporary network failure	Retry with backoff
`429 Too Many Requests`	Rate limit	Wait for reset, then retry
`400 Bad Request`	Malformed payload	Abort and fix the caller
`401 Unauthorized`	Bad or expired credentials	Abort and alert
`403 Forbidden`	Blocked, missing auth, or wrong region	Usually abort, sometimes retry with different routing

The failure mode matters. If your payload is invalid JSON, five retries produce five invalid requests. If credentials expired, retries just delay the alert that someone needs to refresh the session.

Add a circuit breaker per target

Even good retry logic can hurt you when a whole domain changes behavior. Maybe the site moved prices behind client-side rendering. Maybe it started returning login pages. Maybe your parser now throws Unexpected token < in JSON at position 0 because it expected an API response and got HTML.

A small circuit breaker keeps that from taking down the rest of the pipeline:

type BreakerState = {
  failures: number;
  openedUntil?: number;
};

const breakers = new Map<string, BreakerState>();

function canFetch(domain: string, now = Date.now()) {
  const state = breakers.get(domain);
  return !state?.openedUntil || state.openedUntil <= now;
}

function recordFailure(domain: string, now = Date.now()) {
  const state = breakers.get(domain) ?? { failures: 0 };
  state.failures += 1;

  if (state.failures >= 3) {
    state.openedUntil = now + 5 * 60 * 1000;
  }

  breakers.set(domain, state);
}

function recordSuccess(domain: string) {
  breakers.delete(domain);
}

Start with 3 consecutive failures and a 5 minute cool-off. Tune it from production data. The goal is not to avoid every failed request. The goal is to stop one broken target from consuming your whole worker pool.

Store job IDs like production data

If a job matters, persist its ID with the original request, target URL, tenant, attempt count, and deadline. Logs are not enough. When a downstream workflow times out, you want to answer basic questions:

Did we submit the job?
Is it still processing?
Did it fail permanently?
Which tenant or source caused the retry storm?

A simple table is often enough:

create table web_fetch_jobs (
  id text primary key,
  tenant_id text not null,
  target_url text not null,
  status text not null,
  attempts integer not null default 0,
  deadline_at timestamptz not null,
  created_at timestamptz not null default now(),
  updated_at timestamptz not null default now()
);

Once you have this, you can run polling from a queue, recover after deploys, and inspect stuck jobs without guessing.

The practical next step is to audit any live retrieval path that still blocks on a single HTTP request. Replace it with job submission, persisted IDs, bounded polling, and explicit failure classification.

The full breakdown is here if you want the complete picture.

Before you automate a browser, check the network tab

Anakin — Mon, 29 Jun 2026 10:00:01 +0000

You need data from a website, so the first instinct is often Playwright, Puppeteer, Selenium, or an AI browser agent. That works, but it is easy to miss something important: the browser probably did not create the data you want. It fetched it from an internal API and rendered it.

Open DevTools on a product page, job board, travel site, or finance dashboard. Filter the Network tab to fetch or xhr. Reload the page. A lot of the useful stuff is already JSON.

The browser is often the expensive part

A browser automation script has real costs:

Chromium has to start or be kept warm
The page has to load assets you may not care about
Selectors break when the UI changes
Timeouts happen when ads, modals, or lazy loading behave differently
If an LLM agent drives the browser, every step may involve screenshots, DOM snapshots, and model calls

For example, this Playwright script can extract product data from rendered HTML:

import { chromium } from "playwright";

const browser = await chromium.launch();
const page = await browser.newPage();

await page.goto("https://example.com/products/123", {
  waitUntil: "networkidle",
  timeout: 30000,
});

const price = await page.locator("[data-testid='price']").innerText();
const title = await page.locator("h1").innerText();

console.log({ title, price });
await browser.close();

This is fine until the site renames data-testid, renders the price after a delayed client-side request, or shows a cookie modal that covers the page. The failure usually looks boring:

locator.innerText: Timeout 30000ms exceeded.
Call log:
  - waiting for locator('[data-testid="price"]')

That error tells you the selector failed, not whether the data was unavailable. The JSON endpoint may still have returned the price correctly.

Start by finding the data source

Before writing browser automation, capture the request that populated the page.

In Chrome DevTools:

Open Network
Filter by fetch or xhr
Reload the page
Click likely requests
Check Response and Preview for structured JSON
Right click the request and copy as cURL

You might end up with something like this:

curl 'https://www.example.com/api/products/123?currency=USD' \
  -H 'accept: application/json' \
  -H 'user-agent: Mozilla/5.0' \
  -H 'x-client-version: web-2026.06.1'

If that works outside the browser, you can usually replace a browser job with a normal HTTP client:

const res = await fetch("https://www.example.com/api/products/123?currency=USD", {
  headers: {
    accept: "application/json",
    "user-agent": "Mozilla/5.0",
    "x-client-version": "web-2026.06.1",
  },
});

if (!res.ok) {
  throw new Error(`HTTP ${res.status}: ${await res.text()}`);
}

const product = await res.json();
console.log(product.price, product.availability);

That is the clean path: less infrastructure, lower latency, fewer moving parts, and easier retries.

For teams that want maintained access to these private website APIs instead of managing request signatures themselves, Wire exposes cataloged site actions as normal REST calls with structured JSON responses.

Where direct HTTP gets messy

The copied cURL request is often not enough in production.

Many sites include values that change per session or per request:

CSRF tokens
signed query parameters
device or browser fingerprints
short-lived session cookies
nonces that prevent replay
GraphQL operation hashes

The failure mode is usually a 401, 403, or a JSON response that looks valid but contains an application-level denial:

{
  "error": "invalid_signature",
  "message": "request signature expired"
}

This is where a direct network approach stops being just “copy as cURL”. You need to understand how the frontend generates those headers and tokens. Sometimes that is straightforward. Sometimes the signing code changes weekly.

For supported sites, Wire maintains stable action identifiers while handling endpoint changes, signatures, and authenticated session details behind the API boundary.

If you build this yourself, treat the integration like any other external dependency. Add contract tests against the response shape. Alert on 403 spikes. Store HAR files for debugging. Do not assume a private endpoint is stable just because it returned JSON today.

When a browser is still the right tool

Direct HTTP is not always better. It is better when the data already exists as a request you can reproduce reliably.

Use browser automation when the task depends on the visual state of the page:

multi-page forms with conditional fields
portals that require interactive navigation
flows where validation messages affect the next action
sites that render data only after complex client-side state changes
tasks where a human would say “click the second matching result”

This is where tools like Skyvern make sense. Skyvern runs Chromium, observes screenshots and DOM, asks an LLM what to do, and executes actions through Playwright. Its cached runs can replay generated Playwright scripts, which removes the LLM from successful repeat paths, but it still runs a browser.

That distinction matters. A cached browser workflow may avoid model latency, but it still pays for page load, browser memory, navigation, and UI fragility. A live AI browser agent can adapt to UI changes, but it may take many seconds per step and can fail in ways that are harder to reproduce than a bad HTTP response.

A practical decision rule

Use this order:

Check the Network tab first
If the data is available as JSON, try to reproduce the request with curl
If the request needs tokens or signatures, decide whether maintaining that logic is worth it
If the workflow is visual, stateful, or form-heavy, use browser automation
If volume is high, avoid browsers unless there is no reliable network-layer option

The next time you start a scraping or automation task, spend 10 minutes capturing the HAR file before opening Playwright. If you can turn the target action into one authenticated HTTP request with a testable JSON schema, do that first. If you cannot, then reach for the browser with a clearer reason.

How to evaluate a search API for AI agents without confusing search with extraction

Anakin — Wed, 24 Jun 2026 10:00:01 +0000

A lot of AI search integrations look fine in a demo and fall apart in production for a boring reason: search results are not documents. Most search APIs return titles, URLs, snippets, and maybe relevance scores. If your RAG pipeline, research agent, or monitoring job needs full page content, you need a second step. If your agent needs to log in, click through a web app, or act on a platform, you need more than search.

That distinction matters when comparing tools like Anakin and Tavily. Tavily is a focused search and research API. Anakin includes search, extraction, crawling, mapping, browser automation, persistent browser sessions, and platform endpoints. The better choice depends less on who has the nicer search endpoint and more on what your workflow becomes after the first query.

Search results are usually just pointers

A basic search response is useful for ranking candidate sources, not for building a complete context window.

You typically get something like this:

{
  "results": [
    {
      "title": "Pinecone vs Weaviate comparison",
      "url": "https://example.com/vector-db-comparison",
      "snippet": "A comparison of managed and open source vector databases...",
      "score": 0.82
    }
  ]
}

That is not enough if you need to quote the page, chunk it, embed it, or verify claims. You need to fetch and extract the URL afterward.

A safer pipeline looks like this:

async function research(query) {
  const searchResults = await searchApi.search({
    query,
    limit: 10
  });

  const pages = [];

  for (const result of searchResults.results) {
    try {
      const page = await extractionApi.extract({ url: result.url });

      if (!page.text || page.text.length < 500) {
        console.warn("weak extraction", result.url);
        continue;
      }

      pages.push({
        url: result.url,
        title: result.title,
        text: page.text
      });
    } catch (err) {
      console.warn("extract failed", result.url, err.message);
    }
  }

  return pages;
}

The important part is not the exact SDK. It is the control flow: search first, extract second, handle partial failure. Extraction fails in normal cases: JavaScript-heavy pages, bot checks, paywalls, broken HTML, redirects, consent screens, and pages that return a 200 status with almost no meaningful content.

Test the workflow you actually need

If your product only needs public web search, Tavily is a reasonable fit. It has a simple API, relevance scoring, research endpoints, and public features around PII filtering and prompt injection protection. Those request-level safety features matter if your agent feeds search output directly into an LLM.

If your product starts at search but eventually needs platform interaction, the evaluation changes. A recruiting agent might search the web at first, then need LinkedIn company data. A real estate workflow might search market pages, then need Zillow-style structured records. A sales ops agent might research companies, then submit forms or send messages.

If your search pipeline later has to read or act inside platforms without public APIs, Wire is the kind of platform endpoint layer that changes the tool decision from search quality to workflow coverage.

Without that kind of layer, teams usually add one of three things:

A second vendor for platform-specific data
Browser automation with Playwright or Puppeteer
Internal scrapers that someone now has to maintain

None of those are wrong. They just have costs. Browser automation is flexible, but it brings session handling, CAPTCHA risk, proxy management, and brittle selectors. Platform-specific vendors can be reliable, but each one adds another contract, auth model, data shape, and failure mode.

Cost comparisons need the full call count

Search pricing often looks cheaper than it really is because teams count only the first API call.

For example, using the numbers from the Anakin and Tavily public pricing comparison:

Anakin Scale: $100/month for 120,000 credits
Anakin search: 3 credits per query, so about 40,000 searches
Tavily PAYG: $0.008 per credit
Tavily basic search: 1 credit per query

That puts the rough crossover around 12,500 searches per month. Below that, Tavily’s lower-friction usage model can make sense. At 20,000 searches, Tavily PAYG would be about $160 while Anakin Scale stays at $100. At 40,000 searches, Tavily would be about $320 while Anakin remains $100 for search volume within that plan.

But that is only the search call. If every useful result requires extraction, include those calls too. If one user query produces 10 search results and you extract 5 pages, your cost model should count 6 API operations, not 1.

A simple spreadsheet beats guessing:

monthly_user_queries = 8_000
searches_per_query = 1
urls_extracted_per_query = 5

monthly_search_calls = 8_000
monthly_extract_calls = 40_000
total_billable_operations = 48_000

Also account for retries. If 15 percent of extraction calls fail and you retry once, that adds 6,000 more extraction attempts in this example.

Safety features and compliance are not the same thing

Do not treat security checkboxes as interchangeable.

Request-level protections answer questions like:

Does the API filter PII before returning results?
Does it detect prompt injection in retrieved content?
Can I safely pass search output into an LLM without writing my own filters?

Infrastructure and compliance features answer different questions:

Does the vendor have SOC 2 or ISO certification?
How do they store credentials or browser sessions?
Can procurement approve the vendor?
Is tenant isolation documented?

Tavily’s public PII filtering and prompt injection protection are meaningful if you need safety at the search layer. Anakin’s broader stack matters more if your workflow includes crawling, authenticated browsing, or platform actions. Those are different requirements, not a single winner-takes-all category.

A practical evaluation plan

Before choosing a search API, build a small fixture set from your real workload:

20 normal queries
10 obscure or long-tail queries
10 queries where freshness matters
10 pages that require extraction
5 pages likely to fail because of JavaScript, auth, or bot protection

Run each provider through the same script. Store raw responses. Measure:

How many useful sources came back in the top 10
How often extraction returned enough text
Median and p95 latency
Retry rate
Total calls per completed user task
Whether safety filtering happens in the API or in your code

The decision usually becomes obvious after that. Pick the focused search API if your workflow stays search-and-retrieve and its safety features match your requirements. Pick the broader web automation stack if your agent needs authenticated sessions, platform data, browser execution, or write actions.

The next useful step is to write the test harness before wiring either provider into your app. A weekend of fixture-based testing will tell you more than a feature table.

Network-layer scraping is fast, but the maintenance cost is the real decision

Anakin — Tue, 23 Jun 2026 10:00:01 +0000

If you have ever shipped a Puppeteer or Playwright scraper, you know the usual failure mode: it works locally, then production gets slower, a selector changes, a cookie modal appears, or a page loads differently in headless mode. A lot of teams eventually notice that the browser is not where the useful data comes from. The frontend usually calls a JSON endpoint in the background, and that endpoint is what you actually wanted.

The useful trick: replay the request, not the UI

Open DevTools, go to the Network tab, filter by fetch or xhr, then use the site normally. On many modern sites, the page is just a client around internal HTTP, GraphQL, or RPC calls.

For example, a product search page might call something like this:

curl 'https://example.com/api/search?q=headphones&page=1' \
  -H 'accept: application/json' \
  -H 'user-agent: Mozilla/5.0' \
  -H 'x-client-version: web-2026.06.12'

That request might return structured data directly:

{
  "items": [
    { "id": "p_123", "name": "Wireless headphones", "price": 7999, "in_stock": true }
  ],
  "next_page": 2
}

Calling that endpoint from your backend is usually much faster than rendering a page. You avoid DOM waits, image loading, layout work, and most selector problems.

A minimal implementation might look like this:

async function searchProducts(query, page = 1) {
  const res = await fetch(
    `https://example.com/api/search?q=${encodeURIComponent(query)}&page=${page}`,
    {
      headers: {
        accept: 'application/json',
        'user-agent': 'Mozilla/5.0',
        'x-client-version': 'web-2026.06.12'
      }
    }
  );

  if (!res.ok) {
    throw new Error(`Search failed: ${res.status} ${await res.text()}`);
  }

  const body = await res.json();

  if (!Array.isArray(body.items)) {
    throw new Error(`Unexpected schema: ${JSON.stringify(body).slice(0, 300)}`);
  }

  return body.items.map(item => ({
    id: item.id,
    name: item.name,
    priceCents: item.price,
    inStock: item.in_stock
  }));
}

That pattern is simple, cheap, and often good enough. It is also where the real maintenance problem starts.

The endpoint is not a public API

Internal endpoints change without notice. The site owner does not consider your caller a customer of that API.

Common breakages look like this:

401 Unauthorized
{"error":"missing session"}

or:

403 Forbidden
{"message":"invalid client fingerprint"}

or worse, the request still returns 200 OK, but the response shape changes:

{
  "products": [
    { "sku": "p_123", "displayName": "Wireless headphones" }
  ]
}

Your code does not crash at the HTTP layer. It crashes later because body.items is undefined, or it silently writes incomplete data because price moved somewhere else.

This is the tradeoff with network-layer scraping. You remove browser flakiness, but you take ownership of an undocumented API contract.

Generated endpoints are good for exploration

Tools that generate an API from a URL are useful when you need quick coverage. Give the tool a page and a description, wait a short time, and you get an endpoint that extracts fields from the underlying calls.

That is a good fit for:

one-off research
prototypes
read-only extraction
long-tail sites you may never call again
validating whether the target site exposes useful JSON at all

The downside is operational. If the generated endpoint captured /api/v2/search and the site moves to /api/v3/query, someone has to regenerate or repair it. If the site adds a required header, rotates a token format, or changes pagination, your pipeline owns that breakage unless the provider maintains it for you.

For production systems, maintenance matters more than initial generation speed. Wire is one example of the maintained-catalog approach, where the caller keeps using the same action while the provider handles changes in the target site's internal calls.

That model is less flexible for obscure sites, but it fits recurring workflows better.

Read actions and write actions are not the same problem

Reading from an internal endpoint is relatively safe to reason about. You send query params, receive JSON, validate the schema, and store the result.

Writing is harder. A POST request often carries CSRF tokens, session-bound IDs, idempotency keys, hidden form fields, or state derived from previous requests.

A job application flow might look like this:

GET  /jobs/12345
GET  /api/applications/bootstrap?job_id=12345
POST /api/applications/draft
POST /api/applications/draft/resume
POST /api/applications/submit

If you replay only the final POST, you may get:

409 Conflict
{"error":"draft_not_initialized"}

If you reuse an old token, you may get:

422 Unprocessable Entity
{"error":"csrf_token_expired"}

If you retry a non-idempotent action after a timeout, you may submit twice.

This is why read-only extraction and web action automation should not share the same risk model. Reading a price wrong is bad. Sending the same customer message twice is a different class of bug. For workflows that submit, post, message, bid, or apply, Wire treats read and write actions as cataloged operations rather than leaving each caller to reconstruct the browser flow.

Even if you build this yourself, make write actions explicit in your codebase. Do not hide them behind a generic fetchPageData() helper.

What I would test before depending on network replay

For any target site, I would answer these questions before calling it production-ready:

Does the endpoint require an authenticated session?
How long do session cookies or tokens last?
Does the response schema stay stable across pages, accounts, and regions?
What status code appears when the site blocks or challenges the request?
Can the request be retried safely?
Do you have a fixture test that fails when the response shape changes?

A simple schema check catches a lot:

function assertSearchResponse(body) {
  if (!body || !Array.isArray(body.items)) {
    throw new Error('Expected body.items to be an array');
  }

  for (const item of body.items) {
    if (typeof item.id !== 'string') throw new Error('Missing item.id');
    if (typeof item.name !== 'string') throw new Error('Missing item.name');
    if (typeof item.price !== 'number') throw new Error('Missing item.price');
  }
}

Run that against real responses on a schedule. If the test fails before your customer-facing job fails, you bought yourself time.

The practical decision

Use browser automation when you need to interact with a UI that has no useful network calls, or when visual state matters.

Use direct network replay when the site exposes structured background requests and you can tolerate maintaining an undocumented contract.

Use generated endpoints when speed matters more than long-term stability.

Use maintained integrations when the same action runs repeatedly, failure has business impact, or the workflow writes back to the target site.

A good next step is to pick one flaky browser scraper, inspect its XHR calls, replay the smallest useful request with curl, then add a schema test around the response. That will tell you quickly whether network replay reduces the problem or just moves the complexity somewhere else.

Stop scraping the page when the data is already in the network tab

Anakin — Thu, 18 Jun 2026 10:00:04 +0000

You write a scraper with Playwright, wait for the page to load, close the cookie banner, click a filter, and parse a table out of the DOM. Then someone redesigns the page and your selector breaks. The annoying part is that the data probably never lived in the HTML in the first place.

Most modern websites render a UI around structured background requests. The browser loads the shell, runs JavaScript, and calls internal endpoints for prices, availability, inventory, search results, profile data, or whatever the page needs. If you scrape the rendered page, you often process hundreds of kilobytes of layout and tracking code to recover a few kilobytes of JSON.

Look at the network layer before writing browser code

Before reaching for Playwright or Puppeteer, open DevTools and check what the site actually does.

In Chrome:

Open DevTools
Go to the Network tab
Filter by Fetch/XHR
Perform the action manually, such as search, filter, paginate, or change dates
Click the request that returns the data
Inspect the request URL, method, headers, payload, and response

You will often find something like this:

POST /api/search/hotels HTTP/2
content-type: application/json
x-csrf-token: 8f9c...

{
  "checkIn": "2026-03-12",
  "checkOut": "2026-03-15",
  "city": "Berlin",
  "guests": 2
}

And the response is already the thing you wanted:

{
  "results": [
    {
      "id": "hotel_123",
      "name": "Example Hotel",
      "price": 184,
      "currency": "EUR",
      "available": true
    }
  ]
}

At that point, scraping the DOM is extra work. You can reproduce the request directly:

curl 'https://example.com/api/search/hotels' \
  -X POST \
  -H 'content-type: application/json' \
  -H 'x-csrf-token: 8f9c...' \
  --data '{"checkIn":"2026-03-12","checkOut":"2026-03-15","city":"Berlin","guests":2}'

Or from code:

const res = await fetch('https://example.com/api/search/hotels', {
  method: 'POST',
  headers: {
    'content-type': 'application/json',
    'x-csrf-token': process.env.CSRF_TOKEN
  },
  body: JSON.stringify({
    checkIn: '2026-03-12',
    checkOut: '2026-03-15',
    city: 'Berlin',
    guests: 2
  })
});

if (!res.ok) {
  throw new Error(`Search failed: ${res.status} ${await res.text()}`);
}

const data = await res.json();
console.log(data.results.map(h => [h.name, h.price]));

This is the basic pattern: use the browser to discover the request, not to run every request forever.

Wire works in this network-layer space by mapping the background requests a site already uses and exposing stable endpoints around those flows.

Why this matters in production

Browser automation is useful, but it is expensive for data retrieval.

A headless browser has to start a browser process, load HTML, download scripts, execute JavaScript, build a DOM, run layout work, and wait for the app to settle. If you only need the JSON response behind a search form, most of that work is overhead.

The difference shows up quickly:

// Browser-level approach
const page = await browser.newPage();
await page.goto('https://example.com/hotels?q=berlin');
await page.waitForSelector('[data-testid="hotel-card"]');
const prices = await page.$$eval('[data-testid="hotel-card"]', cards =>
  cards.map(card => card.textContent)
);

// Network-level approach
const res = await fetch('https://example.com/api/search/hotels', options);
const { results } = await res.json();
const prices = results.map(h => h.price);

The first version breaks if the frontend team renames data-testid, changes the card layout, lazy-loads the list differently, or inserts an experiment variant. The second version breaks if the internal API changes, auth changes, or the site adds request signing. Neither is magic. The network version just avoids tying your pipeline to visual presentation.

It also gives less junk to downstream systems. If an LLM needs to answer, "Which hotels are available under 200 EUR?", passing it rendered HTML or markdown forces it to ignore navigation, legal text, ads, and duplicated labels. Passing structured JSON lets you validate, filter, and summarize before the model sees anything.

The edge cases are real

Internal endpoints are not public contracts. Treat them as dependencies that can change without notice.

Common failure modes include:

401 or 403 because the request needs a logged-in session
400 because the frontend generated a hidden token you did not include
429 because you ignored rate limits
Empty results because a required header, locale, currency, or experiment flag is missing
A successful 200 response with a changed JSON shape
Requests signed with timestamps or hashes generated in bundled JavaScript

A small amount of defensive code helps:

function parseHotelSearch(payload) {
  if (!payload || !Array.isArray(payload.results)) {
    throw new Error('Unexpected hotel search response shape');
  }

  return payload.results.map(item => {
    if (typeof item.name !== 'string' || typeof item.price !== 'number') {
      throw new Error(`Invalid hotel item: ${JSON.stringify(item)}`);
    }

    return {
      id: item.id,
      name: item.name,
      price: item.price,
      currency: item.currency ?? 'UNKNOWN',
      available: item.available === true
    };
  });
}

You still need monitoring. Log status codes, response sizes, parse failures, and schema changes. If the payload usually contains results and suddenly returns items, you want an alert before a customer notices missing data.

For teams that do not want to own endpoint discovery, request signing, and breakage handling for many third-party sites, Wire is a managed version of this same approach rather than a DOM scraping wrapper.

When a browser is still the right tool

Use browser automation when the browser behavior is the thing you need to test or reproduce.

Good cases for Playwright, Puppeteer, or Selenium:

End-to-end testing user flows
Capturing screenshots or PDFs
Interacting with canvas-heavy or browser-only apps
Debugging frontend behavior
Handling flows where the data is only available after complex client-side state changes
Verifying that the UI actually displays what the API returned

Bad cases:

Polling prices every five minutes
Pulling paginated search results
Checking inventory across many SKUs
Feeding structured records into a data pipeline
Giving an agent live availability data

For those, inspect the network requests first. If the data is already JSON, call that layer directly, validate the response shape, and keep the browser out of the hot path unless you actually need it.

Choosing Between Scraping APIs, Browser Sessions, and Self-Managed Playwright

Anakin — Wed, 17 Jun 2026 10:00:02 +0000

Most teams do not start by designing scraping infrastructure. They start with a script. Then the script becomes a cron job, the cron job becomes a queue, and eventually someone is debugging why Chromium processes are eating all the memory on a worker node. At that point, the question is not “how do we scrape this page?” It is “which parts of this system do we actually want to own?”

The three common options

For continuously updated AI systems, most web data pipelines land in one of three patterns.

Pattern	Best fit	Main cost	Common failure
Async scraping API	Public pages, batch ingestion, RAG refreshes	Per job or per request	Vendor limits, polling complexity
Hosted browser sessions	Authenticated flows, multi-page navigation	Session time	Paying for idle browser minutes
Self-managed Playwright or Puppeteer	High volume, custom browser behavior	Engineering time and infrastructure	Proxy, retry, and browser maintenance

None of these is universally better. The right choice depends on volume, authentication, latency, and how much operational work your team can absorb.

Async APIs: good default for background ingestion

Async APIs work well when you can submit work and collect the result later. That maps cleanly to RAG ingestion, where you usually do not want a user request waiting on a live browser.

The shape is simple:

async function submitUrl(url) {
  const res = await fetch("https://scraper.example.com/jobs", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.SCRAPER_TOKEN}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify({ url, render: true })
  });

  if (!res.ok) throw new Error(`submit failed: ${res.status}`);
  return res.json();
}

You store the returned job ID, poll until completion, validate the result, then update your database or vector store.

The tradeoff is latency. If a job takes 3 to 15 seconds, that is acceptable for background refresh. It is not acceptable inside a chat completion path unless you stream an intermediate answer or tell the user you are fetching fresh data.

Async APIs also introduce rate limits. If the submit endpoint allows 60 requests per minute, this loop will eventually fail:

await Promise.all(urls.map(url => submitUrl(url)));

Use a limiter instead:

import pLimit from "p-limit";

const limit = pLimit(5);

await Promise.all(
  urls.map(url => limit(() => submitUrl(url)))
);

For teams choosing this path, Wire fits the async scraping API category: submit URLs, track job IDs, and consume structured page output without running your own browser pool.

Hosted browser sessions: use them when state matters

Some sources require login, cookies, local storage, or multi-step navigation. A stateless scrape request is a bad fit for that.

Example workflow:

Log in
Navigate to an account page
Open a report
Apply filters
Extract the rendered table

You can do this with a hosted browser session by connecting Playwright to a remote browser endpoint:

import { chromium } from "playwright";

const browser = await chromium.connectOverCDP(process.env.REMOTE_BROWSER_WS);
const page = await browser.newPage();

await page.goto("https://example.com/login");
await page.fill("input[name=email]", process.env.APP_USER);
await page.fill("input[name=password]", process.env.APP_PASSWORD);
await page.click("button[type=submit]");

await page.goto("https://example.com/reports");
await page.click("text=Last 30 days");

const rows = await page.locator("table tbody tr").evaluateAll(nodes =>
  nodes.map(row => row.innerText)
);

await browser.close();

The provider runs the browser. You control the interaction.

This is useful, but the billing model matters. Browser services often charge by session time. A script that waits on slow pages, sleeps between actions, or leaves sessions open after errors can become expensive.

Always close sessions in finally:

let browser;

try {
  browser = await chromium.connectOverCDP(process.env.REMOTE_BROWSER_WS);
  // work
} finally {
  await browser?.close();
}

Hosted browser sessions are a middle ground. You avoid maintaining browser hosts, but you still own workflow logic, selectors, validation, and credentials handling.

Self-managed Playwright: control is not free

Running Playwright yourself gives you the most control. You can install extensions, intercept network calls, tune launch flags, control geographies, and keep browser pools warm.

A basic worker looks harmless:

import { chromium } from "playwright";

const browser = await chromium.launch({ headless: true });

for (const url of urls) {
  const page = await browser.newPage();
  try {
    await page.goto(url, { waitUntil: "networkidle", timeout: 30000 });
    const title = await page.title();
    console.log({ url, title });
  } catch (err) {
    console.error(`failed ${url}:`, err.message);
  } finally {
    await page.close();
  }
}

await browser.close();

Production adds the parts this example avoids:

browser pool sizing
queue backpressure
proxy rotation
retry budgets
per-domain rate limits
CAPTCHA and bot detection handling
memory leak monitoring
selector maintenance
alerting when extraction quality drops

The common mistake is comparing API credits to server cost only. That ignores engineering time. If a site redesign breaks ten selectors and one developer spends a day fixing parsers, that cost belongs in the scraping budget.

Self-managed starts to make sense when volume is high enough, the workflows need custom behavior, and the team already has operational capacity. As a rough heuristic, below hundreds of thousands of pages per month, managed options often cost less once maintenance is included. Above that, run the numbers with your actual failure rate and developer time.

A practical decision process

Ask these questions in order.

Does the source require authentication or multi-step state?

If yes, start with hosted browser sessions or self-managed Playwright. Async single-page extraction can work only if you can provide valid session state safely and the page does not need interaction.

Does the user need the data during the current request?

If yes, avoid long async scraping jobs in the hot path. Use cached content, stream progress, or design the product around delayed retrieval. If no, async jobs are usually simpler.

Do you need custom browser behavior?

If you need extensions, custom TLS behavior, request interception, or nonstandard anti-bot handling, self-managed may be justified. If not, owning that stack is probably unnecessary.

Can you tolerate selector maintenance?

If your extraction depends on CSS selectors, treat redesigns as expected events. Add validation and alerts for empty fields. A scraper that returns 200 OK with empty extracted data has still failed.

if (!result.price || !result.name) {
  await alerts.send({
    source: url,
    reason: "extraction returned missing required fields"
  });
}

The short version

Use async APIs for background ingestion from public pages. Use hosted browser sessions when login and navigation state matter. Use self-managed Playwright when you need control badly enough to pay for it with infrastructure and maintenance.

Do not decide from request volume alone. Include failure rates, selector churn, retry behavior, developer time, and whether stale data causes real product problems.

The full breakdown is here if you want the complete picture.

Async Scraping Jobs Are Usually a Better Fit for RAG Ingestion Than Blocking Requests

Anakin — Tue, 16 Jun 2026 10:00:01 +0000

A RAG system that depends on web data usually fails in a boring way: the page changed, your index did not, and the model answers confidently from stale context. The first fix people reach for is often a cron job plus a scraper. That works until page loads take 12 seconds, a target site returns 429s, and your app starts holding open worker threads for data that should have been fetched in the background.

Why synchronous scraping breaks down

A synchronous scraper looks simple:

const html = await fetchPage(url);
const data = extractProduct(html);
await vectorStore.upsert(data);

That code hides a few production problems.

If fetchPage(url) needs a browser, the request might spend several seconds waiting for JavaScript, network calls, cookie banners, or anti-bot checks. If you run this inside an API request handler, the user waits. If you run enough of them in parallel, your workers sit around holding memory and open sockets.

The failure modes are also awkward:

TimeoutError: Navigation timeout of 30000 ms exceeded
HTTPError: Response code 429 (Too Many Requests)
Error: selector ".price" did not match any elements
ProtocolError: Target closed

Each error needs a different response. A timeout might need a retry. A 429 might need backoff or a different proxy. A missing selector might mean the site redesigned its markup, or it might mean the product is unavailable. Treating all of those as try again later creates bad data and noisy queues.

For AI pipelines, the bigger issue is freshness. If your ingestion job runs hourly, the model can be wrong for up to an hour even when every component is technically healthy. For inventory, shipment status, event schedules, job listings, or pricing, that gap matters.

The async job pattern

Async scraping APIs use a submit, poll, retrieve flow. You submit work, get a job ID, and fetch the result later. The important part is not the API shape itself. The important part is that page retrieval no longer blocks your application flow.

A minimal version looks like this:

# Submit work
curl -X POST https://scraper.example.com/jobs \
  -H "Authorization: Bearer $SCRAPER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/products/123","render":true}'

# Response
# { "jobId": "job_8f31", "status": "queued" }

Then poll by ID:

curl https://scraper.example.com/jobs/job_8f31 \
  -H "Authorization: Bearer $SCRAPER_TOKEN"

# Possible response
# { "jobId": "job_8f31", "status": "running" }

# Completed response
# {
#   "jobId": "job_8f31",
#   "status": "completed",
#   "markdown": "# Product 123\n\nPrice: $42",
#   "html": "<html>...</html>",
#   "durationMs": 8420
# }

In application code, keep polling separate from request handling:

async function waitForScrape(jobId, { timeoutMs = 60000 } = {}) {
  const started = Date.now();
  let delay = 2000;

  while (Date.now() - started < timeoutMs) {
    const res = await fetch(`https://scraper.example.com/jobs/${jobId}`, {
      headers: { Authorization: `Bearer ${process.env.SCRAPER_TOKEN}` }
    });

    if (!res.ok) {
      throw new Error(`status poll failed: ${res.status}`);
    }

    const job = await res.json();

    if (job.status === "completed") return job;

    if (job.status === "failed") {
      throw new Error(`scrape failed: ${job.reason || "unknown reason"}`);
    }

    await new Promise(resolve => setTimeout(resolve, delay));
    delay = Math.min(delay * 1.5, 10000);
  }

  throw new Error(`scrape job ${jobId} timed out after ${timeoutMs}ms`);
}

This gives you a clean boundary. Your ingestion worker can submit 500 URLs, store job IDs, and update records as jobs finish. Your user-facing app does not need to keep browsers open or know how proxy retries work.

For this specific async ingestion pattern, Wire exposes web extraction as jobs that return IDs for polling, which matches the way background RAG updates usually need to run.

What you still need to build

Async jobs do not remove all pipeline work. They move the browser and network mess behind an API, but you still need to decide how your system treats results.

At minimum, store job state in your own database:

create table scrape_jobs (
  id text primary key,
  source_url text not null,
  status text not null,
  submitted_at timestamptz not null default now(),
  completed_at timestamptz,
  error text,
  content_hash text
);

The content_hash matters. If the page content did not change, do not re-embed it and write duplicate vectors. A simple hash over normalized Markdown is often enough:

import crypto from "node:crypto";

function hashContent(markdown) {
  return crypto
    .createHash("sha256")
    .update(markdown.replace(/\s+/g, " ").trim())
    .digest("hex");
}

You also need dead-letter handling. If a job fails three times, stop retrying it in the hot path and send it somewhere visible. Otherwise one broken source can consume your queue forever.

if (job.attempts >= 3) {
  await db.deadLetters.insert({
    sourceUrl: job.sourceUrl,
    reason: job.error,
    failedAt: new Date()
  });
  return;
}

Selector drift is a separate problem

Async scraping helps with latency and reliability, but it does not automatically fix extraction logic. If your parser depends on this:

const price = $(".product-price .amount").text();

then a redesign can silently produce an empty string. That is worse than a hard failure because bad data enters the index.

Prefer extraction contracts that validate output before ingestion:

const ProductSchema = z.object({
  name: z.string().min(1),
  price: z.string().min(1),
  availability: z.enum(["in_stock", "out_of_stock", "unknown"])
});

const parsed = ProductSchema.safeParse(extracted);

if (!parsed.success) {
  throw new Error(parsed.error.message);
}

Whether the extraction comes from selectors, readability output, or an AI extraction step, validate it before it reaches your vector store.

Where async jobs fit

Async scraping jobs fit background ingestion, scheduled refreshes, change detection, and batch updates. They do not fit cases where a chat response must include a page fetched milliseconds ago. A typical scrape job may take several seconds, especially with browser rendering.

If the user can wait, fine. If not, use cached content and update it behind the scenes.

The practical pattern is simple: keep scraping out of request handlers, treat each scrape as a job with state, validate the extracted shape, hash content before embedding, and dead-letter repeated failures instead of retrying forever.

The full breakdown is here if you want the complete picture.

Designing credential handling for authenticated browser automation

Anakin — Fri, 12 Jun 2026 10:00:02 +0000

If you build automation that logs into third-party websites, the dangerous shortcut is obvious: save the username and password, then replay them whenever the job runs. It works, until a log line, API response, support tool, database dump, or overbroad internal permission turns that stored password into a real incident.

Treat the password as a login event, not stored state

A safer model is to treat the password as input to one operation: establishing a session. After login succeeds, store the browser session state, not the password.

The basic flow looks like this:

User enters credentials.
Your service uses them to perform an interactive login.
Your service captures cookies and local storage from the browser context.
Your service encrypts that session state and stores it with tenant and user scope.
Your service discards the password.
Future jobs load the encrypted session state instead of asking for the password again.

Here is a simplified Node.js example using Playwright and AES-256-GCM:

import { chromium } from "playwright";
import crypto from "node:crypto";

function encryptJson(value: unknown, key: Buffer) {
  const iv = crypto.randomBytes(12);
  const cipher = crypto.createCipheriv("aes-256-gcm", key, iv);

  const plaintext = Buffer.from(JSON.stringify(value), "utf8");
  const ciphertext = Buffer.concat([
    cipher.update(plaintext),
    cipher.final()
  ]);

  return {
    ciphertext: ciphertext.toString("base64"),
    iv: iv.toString("base64"),
    tag: cipher.getAuthTag().toString("base64")
  };
}

async function connectAccount({
  tenantId,
  userId,
  username,
  password,
  encryptionKey
}: {
  tenantId: string;
  userId: string;
  username: string;
  password: string;
  encryptionKey: Buffer;
}) {
  const browser = await chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();

  try {
    await page.goto("https://example.com/login");
    await page.fill("input[name=email]", username);
    await page.fill("input[name=password]", password);
    await page.click("button[type=submit]");

    await page.waitForURL("**/dashboard", { timeout: 15000 });

    const storageState = await context.storageState();
    const encrypted = encryptJson(storageState, encryptionKey);

    const sessionId = await db.sessions.insert({
      tenantId,
      userId,
      ciphertext: encrypted.ciphertext,
      iv: encrypted.iv,
      tag: encrypted.tag,
      status: "active",
      expiresAt: new Date(Date.now() + 90 * 24 * 60 * 60 * 1000)
    });

    return { sessionId, status: "connected" };
  } finally {
    await browser.close();
  }
}

Do not overstate what this does. Setting a JavaScript variable to undefined does not guarantee memory erasure. The important design choice is that you never write the password to disk, never return it from an API, never log it, and never keep it as reusable application state.

Sessions are credentials too

A browser session can contain cookies, JWTs, refresh tokens, CSRF tokens, and site-specific state in local storage. If someone steals it, they may not need the password.

So session storage needs nearly the same controls as password storage:

CREATE TABLE browser_sessions (
  id uuid PRIMARY KEY,
  tenant_id uuid NOT NULL,
  user_id uuid NOT NULL,
  ciphertext text NOT NULL,
  iv text NOT NULL,
  auth_tag text NOT NULL,
  status text NOT NULL CHECK (status IN ('active', 'revoked', 'expired')),
  created_at timestamptz NOT NULL DEFAULT now(),
  expires_at timestamptz NOT NULL,
  revoked_at timestamptz
);

CREATE INDEX browser_sessions_tenant_id_idx
ON browser_sessions (tenant_id, user_id, status);

Every read should include tenant scope. Do not fetch by session ID alone and then check ownership later. That creates room for mistakes in future call sites.

SELECT *
FROM browser_sessions
WHERE id = $1
  AND tenant_id = $2
  AND user_id = $3
  AND status = 'active'
  AND expires_at > now()
  AND revoked_at IS NULL;

This is also where API shape matters. Return handles and metadata, not secrets:

{
  "sessionId": "6b9f...",
  "status": "active",
  "expiresAt": "2026-03-01T00:00:00Z"
}

Do not return encrypted session blobs to the frontend “because they are encrypted.” If the client does not need the value, the API should not expose it.

For authenticated browser jobs, Wire follows this session-based model: the automation runs against an encrypted browser session rather than keeping a reusable site password around.

Expiry and revocation need first-class behavior

Sessions expire. Sites rotate cookies. Users change passwords. MFA prompts appear. A good system treats these as normal states, not weird failures.

Bad behavior looks like this:

Error: Timeout 30000ms exceeded

That tells the user nothing. Did the page load slowly? Did login expire? Did the selector change? Did MFA block the run?

Better behavior distinguishes the failure:

if (page.url().includes("/login")) {
  throw new SessionExpiredError({
    code: "SESSION_RECONNECT_REQUIRED",
    message: "The saved browser session is no longer valid. Reconnect the account."
  });
}

Revocation should also take effect before the next job starts, not eventually after some background cleanup runs:

UPDATE browser_sessions
SET status = 'revoked', revoked_at = now()
WHERE id = $1
  AND tenant_id = $2
  AND user_id = $3;

Then your job runner must refuse to load revoked sessions. This sounds obvious, but many systems implement revocation only in the UI and forget the worker path.

Logs are the easiest place to leak secrets

Even if the database design is good, logs can undo it. The common mistake is logging request bodies during login debugging:

logger.info({ body: req.body }, "connect account request");

That can put passwords, cookies, or tokens into your observability system. Use structured redaction by default:

import pino from "pino";

export const logger = pino({
  redact: {
    paths: [
      "req.headers.authorization",
      "req.body.password",
      "password",
      "cookies",
      "storageState.cookies",
      "storageState.origins[*].localStorage[*].value"
    ],
    censor: "[REDACTED]"
  }
});

Also check error reporting. Browser automation failures often include screenshots, HTML snippets, network headers, and console logs. Any one of those can contain account data.

Isolate the browser runtime

Do not reuse one persistent browser profile across users or tenants. Create a fresh context for each job, load only the specific session state needed for that job, then tear it down.

const context = await browser.newContext({ storageState });
try {
  await runAutomation(context);
} finally {
  await context.close();
}

This does not solve every problem. The target site can still rate limit you. MFA can still interrupt execution. A compromised automation script can still misuse the access it has during the run. But isolation reduces accidental cross-account state bleed, which is one of the nastier classes of bugs in browser automation systems.

Wire runs authenticated tasks in scoped browser environments, which addresses the specific risk of one customer’s browser state carrying into another job context.

A practical checklist

If you are reviewing or building this kind of system, trace one account connection from end to end and answer these questions:

Is the password ever written to disk?
Can any API response include a cookie, token, or encrypted session blob?
Do all session reads include tenant and user scope?
What does the worker do when the site redirects to login?
Does revocation block future runs immediately?
Are request bodies, screenshots, HTML dumps, and network headers redacted?
Does each job get a fresh browser context?

Start by testing the failure paths. Expire a session manually, revoke one while a job is queued, and search your logs for the test password. That will tell you more than the happy path ever will.

A safer pattern for authenticated automation: vault-backed sessions

Anakin — Thu, 11 Jun 2026 10:00:01 +0000

Authenticated automation usually fails at the same boring place: login. Scraping a public page is easy compared with running a workflow inside a supplier portal, admin dashboard, CRM, or internal tool. At some point your automation needs a password, and the naive answer is to store that password in your app. That works until you need rotation, revocation, audit logs, or a security review.

A better pattern is to treat the password as a just-in-time input and the session as the reusable artifact.

The pattern: read the secret once, keep the session

For browser automation, you usually do not need the password on every run. You need it when no valid session exists.

The flow looks like this:

Store the password in a vault your team already controls, such as 1Password, Bitwarden, Vault, or AWS Secrets Manager.
Give the automation service a scoped token that can read only the required item.
When the session is missing or expired, read the credential into memory.
Log in with Playwright, Puppeteer, Selenium, or your browser driver of choice.
Persist the resulting session state.
Drop the password from memory and never write it to your database or logs.

Wire applies this model to authenticated browser automations by reading from 1Password when it needs to refresh a login, then running future tasks from the saved session rather than from the password.

The important distinction is this: your application stores something revocable and scoped, not the user's actual credential.

What this looks like in code

Here is a simplified Playwright example. It uses the 1Password CLI to fetch a login item, but the shape is the same if you use a vault SDK or a cloud secrets manager.

import { chromium } from 'playwright';
import { execFile } from 'node:child_process';
import { promisify } from 'node:util';
import fs from 'node:fs/promises';

const execFileAsync = promisify(execFile);

async function readLoginFrom1Password(itemId: string) {
  const { stdout } = await execFileAsync('op', [
    'item',
    'get',
    itemId,
    '--format',
    'json'
  ], {
    env: {
      ...process.env,
      OP_SERVICE_ACCOUNT_TOKEN: process.env.OP_SERVICE_ACCOUNT_TOKEN
    }
  });

  const item = JSON.parse(stdout);

  const username = item.fields.find((f: any) => f.id === 'username')?.value;
  const password = item.fields.find((f: any) => f.id === 'password')?.value;

  if (!username || !password) {
    throw new Error('Vault item is missing username or password');
  }

  return { username, password };
}

async function loginAndSaveSession(itemId: string, sessionPath: string) {
  const { username, password } = await readLoginFrom1Password(itemId);

  const browser = await chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();

  await page.goto('https://example.com/login');
  await page.fill('input[name="email"]', username);
  await page.fill('input[name="password"]', password);
  await page.click('button[type="submit"]');

  await page.waitForURL('https://example.com/dashboard', { timeout: 30_000 });

  await context.storageState({ path: sessionPath });
  await browser.close();
}

async function runAuthenticatedTask(sessionPath: string) {
  const browser = await chromium.launch();
  const context = await browser.newContext({ storageState: sessionPath });
  const page = await context.newPage();

  await page.goto('https://example.com/dashboard');

  if (page.url().includes('/login')) {
    throw new Error('Session expired or rejected');
  }

  // Continue with the authenticated workflow.

  await browser.close();
}

In production, I would avoid shelling out if a maintained SDK exists, but the example makes the boundary clear. The password enters memory for one login operation. The durable artifact is the browser storage state, not the password.

For Playwright, storageState usually contains cookies and localStorage. Treat it like a credential. Encrypt it at rest, restrict access to it, and expire it when the user revokes access.

The failure modes matter

This pattern does not remove authentication problems. It moves them to places you can reason about.

If the vault token is revoked, the next login refresh should fail clearly:

Error: failed to read credential from vault: access denied

Existing sessions may continue until the target site expires them. That is normal browser behavior. If you need immediate cutoff, delete the saved session state when vault access is revoked.

If the password rotates, the next login refresh should pick up the new value automatically. If you copied the password into your own database, rotation becomes a coordination problem. If you read from the vault at login time, rotation stays where it belongs.

If the target site changes its login page, your automation still breaks. You will see symptoms like:

TimeoutError: page.waitForURL: Timeout 30000ms exceeded

or:

Error: Session expired or rejected

That failure has nothing to do with secret storage. You still need monitoring around login success rate, selector failures, MFA challenges, and redirects to unexpected pages.

Audit logs should live outside your app too

One underrated benefit of vault-backed auth is that the vault can log every read. That gives your security team an audit trail in a system they already trust.

Your app should still log its own authentication events:

{
  "event": "credential_read_for_login",
  "vault": "1password",
  "item_id": "supplier-portal-prod",
  "workflow_id": "daily-inventory-sync",
  "timestamp": "2026-06-08T10:15:30Z"
}

Do not log usernames, passwords, session cookies, headers, or form payloads. Item IDs and workflow IDs are usually enough to debug access without leaking the secret itself.

Wire also relies on the vault provider's own activity log for credential reads, which means the audit trail does not only exist inside the automation platform.

What to store and what not to store

A reasonable implementation stores:

Encrypted session cookies or browser storage state
A pointer to the vault item
A scoped vault access token, encrypted separately
Metadata such as status, timestamps, and last refresh result

It should not store:

The password
A copy of the vault item contents
Screenshots of login pages containing filled password fields
Request logs containing auth headers or cookies
Full browser traces from login unless you scrub them carefully

The awkward part is the scoped vault token. It is still sensitive. If someone steals it, they may be able to read the same vault items your automation can read. Limit its scope, store it with envelope encryption, rotate it, and make revocation part of your offboarding process.

A practical way to start

If you already run authenticated automation, inspect where credentials live today. Search your database schema, environment variables, logs, and browser trace storage. Then change one workflow so it reads the password from a vault only when the saved session has expired. Keep the session encrypted, add a clear Session expired or rejected path, and verify that rotating the password in the vault does not require a deploy.

Why Cloudflare Breaks Proxy-Only Scrapers

Anakin — Wed, 10 Jun 2026 10:00:01 +0000

You rotate residential proxies, set a Chrome user agent, and still get a 403. Or worse, you get a 200 response that contains a Cloudflare challenge page instead of the product, article, or search result you expected. That usually means the site is not blocking just your IP address.

The proxy is only one part of the fingerprint

A proxy changes where the request comes from. It does not make your HTTP client behave like Chrome.

This is the common version of the problem:

import requests

url = "https://example.com/product/123"

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/121.0.0.0 Safari/537.36"
    )
}

proxies = {
    "https": "http://user:pass@residential-proxy.example:8000"
}

r = requests.get(url, headers=headers, proxies=proxies, timeout=30)

print(r.status_code)
print(r.text[:500])

The request might return:

403
<!DOCTYPE html><html><head><title>Just a moment...</title>

Or it may return 200, but the body still contains cf-chl, turnstile, or Just a moment. If your scraper only checks the status code, it will treat a block page as a successful scrape.

The user agent header says Chrome, but the rest of the connection does not. Cloudflare can inspect details such as:

TLS Client Hello ordering
Cipher suites and extensions
HTTP/2 SETTINGS frames
Header ordering and casing
JavaScript API behavior
Canvas, WebGL, and font fingerprints
Cookie and session consistency across requests

Python requests, Go net/http, and many scraping libraries do not emit the same low-level network fingerprint as a real browser. A residential IP does not fix that mismatch.

Cloudflare has different layers

It helps to separate the protection types, because they fail in different ways.

Basic challenge pages

These are the interstitial pages that run JavaScript before allowing the request through. A plain HTTP client usually gets stuck here because it does not execute the challenge.

Symptom: HTML contains Just a moment, cf-browser-verification, or challenge scripts instead of the target content.

Turnstile

Turnstile is Cloudflare's challenge widget. Sometimes it runs invisibly. Sometimes it requires interaction.

Symptom: the page renders, but the content remains gated until a token appears or a form submission completes.

Bot Management

Bot Management is the harder case. It can block before your scraper reaches useful HTML because the browser, TLS, protocol, and behavior signals do not line up.

Symptom: rotating IPs does not improve success rate much, and failures happen early with 403s, redirects, or repeated challenge loops.

This is where a browser-rendering approach often becomes necessary, not because browsers are magic, but because the target expects a real browser-shaped session.

Test for content, not just status codes

A basic verification loop should check three things:

Did the request return a 200?
Does the body contain the content you need?
Does the body contain signs of a block page?

For example:

from bs4 import BeautifulSoup

BLOCK_MARKERS = [
    "Just a moment",
    "cf-chl",
    "cf-browser-verification",
    "turnstile",
    "Attention Required!",
]

def classify_response(html: str, expected_selector: str) -> str:
    if any(marker in html for marker in BLOCK_MARKERS):
        return "blocked"

    soup = BeautifulSoup(html, "html.parser")
    if soup.select_one(expected_selector):
        return "ok"

    return "unknown"

result = classify_response(r.text, ".product-title")
print(result)

The unknown case matters. It catches layout changes, partial responses, empty states, and soft blocks that do not use obvious Cloudflare text.

Run this against 3 to 5 representative URLs before scaling. Include the pages that usually fail: product detail pages, search results, login-gated pages, and paginated flows. Ten successful requests against a homepage do not tell you much.

When proxies are still enough

Proxy rotation still works when the site mostly enforces IP reputation or rate limits. If you scrape static pages, public listings, or lightly protected endpoints, a browser may be unnecessary overhead.

The tradeoff is simple:

Proxy-only scraping is faster and cheaper
Browser rendering handles more protection layers
Browser sessions cost more and add latency
CAPTCHA and login flows need explicit handling
Multi-page workflows need cookie and storage persistence

The mistake is treating proxies and browsers as interchangeable. They solve different parts of the fingerprint.

When you have targets with mixed protection levels, Wire provides browser rendering, proxy-based scraping, and structured extraction behind one scraping API, so the mode can match the actual failure pattern instead of forcing every page through the same path: Wire.

A practical rule of thumb

Start with the cheapest thing that can prove it returns real content.

Use HTTP plus proxies if the target returns stable HTML and your validation passes. Move to browser rendering when you see challenge pages, Turnstile, session-sensitive navigation, or failures that do not improve with IP rotation. Keep the validation code in production either way, because a 200 response with a challenge page is still a failed scrape.

The full breakdown is here if you want the complete picture.

How Scraping API Pricing Changes Once You Need Browser Sessions

Anakin — Mon, 08 Jun 2026 10:00:01 +0000

A scraping job that looks cheap in development can get expensive in production. The usual surprise is not the price of one request. It is the number of billable attempts required to finish one logical job, especially when Cloudflare or another bot protection layer forces you into browser rendering.

Per-request billing is easy to reason about

Per-request pricing charges for each URL fetch. If you scrape one page and get one response, the model is clean.

For example, assume a provider charges $0.065 per rendered request:

function perRequestCost({ pages, pricePerRequest }) {
  return pages * pricePerRequest;
}

console.log(perRequestCost({
  pages: 10_000,
  pricePerRequest: 0.065
}));

// 650

That is predictable. It also scales linearly with page count.

This works well for jobs like:

Fetch one article URL
Render one search result page
Take one screenshot
Extract one public profile

It works less well when one job contains many pages from the same site. If you need 50 pages behind the same Cloudflare session, billing each page independently can waste the session state you already paid to establish.

Browser sessions change the unit of work

A browser session has startup cost. It needs to launch, negotiate protection, store cookies, run JavaScript, and keep state. Once that session exists, the next page in the same workflow may be much cheaper than starting over.

Session-duration billing charges for time instead of page count. For example, imagine a browser provider bills one credit per two-minute interval, rounded up.

function sessionCost({ sessions, minutesPerSession, creditsPerInterval, intervalMinutes }) {
  const intervalsPerSession = Math.ceil(minutesPerSession / intervalMinutes);
  return sessions * intervalsPerSession * creditsPerInterval;
}

const cost = sessionCost({
  sessions: 200,
  minutesPerSession: 10,
  creditsPerInterval: 1,
  intervalMinutes: 2
});

console.log(cost);
// 1000 credits

If each 10-minute session scrapes 50 pages, those 200 sessions cover 10,000 pages. The important metric becomes pages per session, not just requests per page.

That model rewards batching. It punishes slow navigation, unnecessary waits, and workflows that open a new browser for every URL.

Wire uses browser-session billing for browser jobs and exposes the work as async jobs, which fits long-running Cloudflare workflows where you care about job completion, retries, and session state rather than a single HTTP response: Wire.

Failed attempts are part of the cost

Pricing pages often show the happy path. Production scrapers spend money on failures too.

You need to know whether the provider charges for:

403 responses
Cloudflare challenge pages
Browser timeouts
CAPTCHA failures
Retries
Empty responses
Pages where extraction fails after rendering succeeds

A failed scrape is not always a failed HTTP request. This response is technically successful from an HTTP perspective:

HTTP/2 200
content-type: text/html

<html>
  <title>Just a moment...</title>
  <script src="/cdn-cgi/challenge-platform/..."></script>
</html>

If your billing model charges per response, you may pay for that. If your extraction pipeline treats it as success, you may also store bad data.

Track cost per successful extraction, not cost per request.

function costPerSuccessfulExtraction({ totalCost, successfulExtractions }) {
  if (successfulExtractions === 0) return Infinity;
  return totalCost / successfulExtractions;
}

console.log(costPerSuccessfulExtraction({
  totalCost: 650,
  successfulExtractions: 8200
}));

// 0.07926829268292683

That number is more useful than advertised request price because it includes blocks, retries, and parsing failures.

What to measure before choosing a model

Run the same small benchmark against each provider you are considering. Use real target URLs, not simple test pages.

For each run, record:

Number of attempted pages
Number of pages with expected content
Number of challenge or block pages
Median and p95 latency
Total billed units
Cost per successful extraction
Whether cookies and local storage persisted across pages

A simple CSV is enough:

provider,mode,pages,successes,blocks,p95_ms,billed_units,total_cost
provider_a,per_request,100,83,17,9400,100,6.50
provider_b,browser_session,100,96,4,18200,12,unknown

The p95 matters because browser rendering can be much slower than proxy-only scraping. If your job runs inside a queue with timeouts, a technically successful browser scrape may still fail your workflow.

Choosing the cheaper option

Use per-request billing when each page stands alone and protection is light. It is simple, and simple is good when the workload fits.

Use browser-session billing when you need to keep state across many pages: authenticated dashboards, carts, search pagination, product catalogs, or sites that challenge new sessions aggressively.

Use proxy bandwidth or enterprise proxy pricing when your team already manages crawling infrastructure and mainly needs IP diversity. The tradeoff is that bandwidth pricing can be harder to predict because page size, compression, retries, and geography all affect the bill.

Before you commit, run a 100-page test and calculate cost per successful extraction. That one number will tell you more than the pricing table.

The full breakdown is here if you want the complete picture.