DEV Community: Anakin

How to benchmark scraping APIs for AI agent workloads

Anakin — Wed, 19 Aug 2026 10:00:02 +0000

Your agent calls fetch() on a pricing page, gets a 200, passes the HTML to the model, and the answer is wrong. Not because the model hallucinated, but because the HTML was mostly an empty React shell, a bot challenge, or a region-specific placeholder. The request succeeded at the HTTP layer and failed at the data layer.

That distinction matters when you evaluate scraping APIs. A status code is not enough. You need to test whether the content your agent needs actually came back.

The failures that matter

Most scraping failures in agent pipelines come from three places.

Client-rendered pages

A lot of docs, pricing pages, and product listings render meaningful content after JavaScript runs. A plain request gets something like this:

<div id="__next"></div>
<script src="/_next/static/chunks/main.js"></script>

That is technically HTML, but it is not useful context for an agent. If your extraction layer sends that to an LLM, the model has to guess.

Anti-bot systems

Cloudflare, Akamai, DataDome, and similar systems do not just check headers. They look at IP reputation, TLS fingerprints, browser behavior, request timing, cookies, and challenge completion. Cloud function IP ranges often get blocked before your code sees anything useful.

The symptom is usually one of these:

403 Forbidden
429 Too Many Requests
Access Denied
Just a moment...

Sometimes it is worse: you get a 200 response containing the challenge page, then your parser happily extracts the wrong text.

Session-dependent content

Some pages need login state, location, cart context, or age gates. A stateless HTTP API request cannot infer those. You need session persistence or a browser you can control.

A better benchmark than “did it return 200?”

A useful benchmark should score content quality, not just request success. The original test behind this article used 24 URLs across static HTML, JS-heavy SPAs, Cloudflare-protected pages, Akamai retail pages, ecommerce, and news/media.

The scoring idea was simple:

0.5 if the response succeeded and returned more than 500 bytes
0.3 if expected content appeared
0.2 if the response was larger than 1,000 bytes
1.0 counted as a pass

You can implement that kind of check without much code:

const tests = [
  {
    url: "https://example.com/pricing",
    mustContain: ["Pricing", "Enterprise"]
  },
  {
    url: "https://example.com/docs",
    mustContain: ["API", "Authentication"]
  }
];

function scoreResponse({ status, body }, expectedTerms) {
  let score = 0;

  if (status >= 200 && status < 300 && body.length > 500) {
    score += 0.5;
  }

  const lower = body.toLowerCase();
  const hasExpectedContent = expectedTerms.some(term =>
    lower.includes(term.toLowerCase())
  );

  if (hasExpectedContent) {
    score += 0.3;
  }

  if (body.length > 1000) {
    score += 0.2;
  }

  return score;
}

for (const test of tests) {
  const res = await scrapeWithProvider(test.url);
  const score = scoreResponse(res, test.mustContain);

  console.log({
    url: test.url,
    status: res.status,
    bytes: res.body.length,
    score,
    pass: score === 1
  });
}

This catches the common false positive: a provider returns a valid HTTP response, but the body contains a login wall, bot challenge, or empty app shell.

What the June 2026 benchmark showed

Six scraping APIs were tested across 24 URLs: Anakin, Firecrawl, Tavily, ScrapingBee, ZenRows, and ScraperAPI. Anakin built and ran the benchmark, so treat it as vendor-produced data. The methodology and raw JSON were published separately, which is the right way to make this kind of comparison inspectable.

The high-level result:

Tool	Passes	Success rate
Anakin	18/24	75%
Firecrawl	17/24	71%
Tavily	15/24	62%
ScrapingBee	14/24	58%
ZenRows	11/24	46%
ScraperAPI	10/24	42%

The category breakdown is more useful than the overall score.

Cloudflare was not equally hard for every tool. Anakin and Firecrawl both passed 5/5 Cloudflare-protected pages. Tavily and ScrapingBee passed 3/5. ZenRows and ScraperAPI passed 2/5.

JS-heavy SPAs were easier. Every tested tool except ScraperAPI passed all 5 SPA tests. That suggests modern scraping APIs mostly know how to run a browser now. The harder question is whether they can keep getting real content once anti-bot systems get involved.

Akamai retail pages were the wall. No tool passed more than 1/3. Firecrawl passed 1/3, and the rest passed 0/3. If your agent depends on large retail sites, you should test those exact domains before committing to any provider.

Ecommerce was also uneven. Firecrawl passed 0/4 there despite doing well on Cloudflare and SPAs. Anakin and ScrapingBee passed 2/4. Tavily, ZenRows, and ScraperAPI passed 1/4.

For repeated extraction from fixed targets like product catalogs or public profiles, Wire uses site-specific endpoints that return structured data instead of rendering pages and fighting the same anti-bot layer on every request.

Latency can break the agent even when scraping works

Agents often run scraping inside a loop: search, open page, extract, reason, open another page. A 40 second scrape is not just slow. It changes the architecture.

In the benchmark, Tavily averaged around 2 seconds. Firecrawl averaged around 3.1 seconds. ScrapingBee was around 5.7 seconds. Anakin averaged 6.3 seconds, including async polling overhead. ZenRows was around 9.2 seconds.

ScraperAPI was the outlier with JavaScript rendering enabled: 43 seconds on average across all 24 requests, with 19 of 24 taking more than 30 seconds. That may be acceptable for offline batch jobs, but it is painful in an interactive agent loop.

Async APIs are not automatically worse. If your agent can submit a job, do other work, then poll later, async is fine. If your control flow blocks on every scrape before taking the next action, synchronous latency matters more.

How I would choose

Start with your URL list, not a vendor feature matrix. Put 20 to 50 real targets into a small benchmark and include expected strings for each page. Test at the same time of day, with the same rendering settings, and record status, bytes, latency, and content match.

Use static pages to establish a baseline, but do not stop there. Include:

5 JS-rendered pages
5 Cloudflare-protected pages
ecommerce or marketplace pages if your product needs them
login-gated pages if sessions matter
a few pages you know are painful

If Cloudflare is the main problem, Anakin and Firecrawl had the strongest result in this dataset. If speed matters more than coverage and volume is low, Tavily is worth testing. If cost matters and the pages are not heavily protected, ScrapingBee may be enough. If you need large volumes from a known set of sites, Wire is relevant because the extraction problem becomes structured endpoint access rather than browser automation.

The practical next step: build a provider-agnostic test harness around your real URLs, score actual content, and keep the raw responses. The failed bodies will tell you more than any pricing page or feature checklist.

Why Scrapers Still Hit HTTP 429 After Adding Delays

Anakin — Tue, 18 Aug 2026 10:00:01 +0000

You add sleep(5) between requests, maybe add random jitter, and the scraper still gets HTTP 429 Too Many Requests. The usual mistake is assuming the server only counts timestamps. Many sites count requests against an identity: IP address, session cookie, account token, browser fingerprint, or some combination of those.

A delay only changes one signal

A basic scraper failure often looks like this:

GET /products/123 -> 200
GET /products/124 -> 200
GET /products/125 -> 200
GET /products/126 -> 429 Retry-After: 120

If those requests all share the same identity, spreading them out may not help much. A sliding-window limiter might allow 3 requests per 2 minutes per fingerprint. Your 5-second delay still puts all 4 requests inside the same window.

The same thing happens when you rotate proxies but reuse the rest of the client state. From your side, each request uses a new IP. From the server side, the requests may still share a cookie, TLS signature, user-agent profile, timezone, header order, or localStorage identifier.

For scraping systems where reliable extraction depends on managing IPs, browser fingerprints, and failure handling together, Wire treats those identity details as part of the extraction path rather than something bolted onto a sleep() loop.

What servers usually bucket

Rate limiting usually uses one of two algorithms:

Sliding window: count requests during the last N seconds.
Token bucket: refill a fixed number of request tokens over time.

The algorithm matters, but the key detail is the bucket key. Servers rarely keep only one global counter. They can count per:

IP address
authenticated account
API key
session cookie
browser fingerprint
TLS fingerprint
device or tracking ID stored in localStorage

That means this change may not fix anything:

import time
import requests

for url in urls:
    response = requests.get(url, proxies=next_proxy())
    print(response.status_code, url)
    time.sleep(10)

If requests.get() always sends the same headers, carries the same cookies, or negotiates the same TLS fingerprint, the server can still group the traffic. Worse, if you persist a browser profile between proxy changes, cookies and localStorage can link every new IP back to the same previous session.

Your retry loop should honor Retry-After

When you do get a 429, first read the response. Many servers send a Retry-After header. It can be an integer number of seconds or an HTTP date.

import random
import time
import requests
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime


def retry_after_seconds(value):
    if not value:
        return None

    try:
        return max(0, int(value))
    except ValueError:
        retry_at = parsedate_to_datetime(value)
        if retry_at.tzinfo is None:
            retry_at = retry_at.replace(tzinfo=timezone.utc)
        return max(0, (retry_at - datetime.now(timezone.utc)).total_seconds())


def get_with_429_backoff(url, session, max_attempts=5):
    for attempt in range(max_attempts):
        response = session.get(url, timeout=30)

        if response.status_code != 429:
            response.raise_for_status()
            return response

        wait = retry_after_seconds(response.headers.get('Retry-After'))
        if wait is None:
            wait = min(60, 2 ** attempt) + random.uniform(0, 1)

        time.sleep(wait)

    raise RuntimeError(f'still rate limited after {max_attempts} attempts: {url}')

This fixes the obvious bad behavior: retrying immediately and making the block worse.

It does not fix identity-based limits. If every retry uses the same cookie jar and browser fingerprint, the server still sees one actor waiting and trying again. Sometimes that is fine. Sometimes the bucket never drains enough for your workload.

Fix the identity lifecycle

If the limit follows the session, manage the session explicitly. Do not let browser state leak across unrelated proxy identities.

With Playwright, that usually means creating isolated contexts and keeping the browser profile internally consistent:

from playwright.async_api import async_playwright


async def fetch_page(url, proxy_url, profile):
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            proxy={'server': proxy_url},
            headless=True,
        )

        context = await browser.new_context(
            user_agent=profile['user_agent'],
            locale=profile['locale'],
            timezone_id=profile['timezone'],
            viewport=profile['viewport'],
        )

        page = await context.new_page()
        response = await page.goto(url, wait_until='domcontentloaded')
        html = await page.content()

        await context.close()
        await browser.close()

        return response.status, html

The important part is not randomizing everything on every request. That can create impossible browser combinations, such as a mobile user agent with a desktop viewport and the wrong timezone for the proxy location. Keep a coherent profile for a session, then discard the context when that session is done.

For repeated extraction against the same URL set, Wire handles job-style URL extraction with retries and cached results, which avoids hitting the origin again for identical work when a cached response is valid.

How to debug the real cause

Do not guess. Log the identity signals you control:

print({
    'status': response.status_code,
    'retry_after': response.headers.get('Retry-After'),
    'proxy_id': proxy_id,
    'user_agent': session.headers.get('User-Agent'),
    'cookie_count': len(session.cookies),
})

Then run small tests:

Same session, different IP: if 429 continues, the limit is probably not only per-IP.
New session, same IP: if it clears, cookies or localStorage were likely involved.
Same headers across many proxies: if all proxies fail together, check fingerprinting.
Honor Retry-After: if that fixes it, you had a timing problem, not an identity problem.

The practical next step is to add this logging around your current scraper, then change one variable at a time: IP, cookie jar, browser context, and retry timing. If you change all of them at once, you may get past the 429, but you still will not know which part was broken.

Testing That Secrets Cannot Reach Logs in Go

Anakin — Mon, 17 Aug 2026 10:00:01 +0000

Nobody plans to log a password. It usually happens when someone is debugging a production issue, prints a struct to see what is going on, fixes the bug, and leaves the log line behind. The review misses it because the change is small and the struct looks harmless. Then your log pipeline does exactly what you asked it to do: collect, index, replicate, and retain the secret.

Policies help, but they do not catch this class of mistake reliably. A better control makes the unsafe thing hard to express and tests the paths where it can still happen.

Redaction has to cover more than String()

In Go, the first version often looks like this:

type DBConfig struct {
    Host     string
    User     string
    Password string
}

func (c DBConfig) String() string {
    return fmt.Sprintf("DBConfig{Host:%q User:%q Password:[REDACTED]}", c.Host, c.User)
}

That helps for this case:

fmt.Printf("%v\n", cfg)

But it does not cover every route to a log line. This leaks the password:

fmt.Printf("%#v\n", cfg)
// main.DBConfig{Host:"db.internal", User:"app", Password:"correct-horse"}

%#v prints a Go-syntax representation. If your redaction only lives in String(), you have not covered that path.

The safer pattern is to make the secret field itself refuse to render as cleartext:

type Secret string

func (Secret) String() string {
    return "[REDACTED]"
}

func (Secret) GoString() string {
    return "[REDACTED]"
}

func (Secret) MarshalJSON() ([]byte, error) {
    return []byte(`"[REDACTED]"`), nil
}

type DBConfig struct {
    Host     string `json:"host"`
    User     string `json:"user"`
    Password Secret `json:"password"`
}

In Wire, we use this kind of type-level redaction for values that may hold vault tokens, resolved credentials, or connection material, because the debug line is the realistic failure mode.

This is still not magic. If someone converts Secret back to string, they can leak it. If a logging library uses reflection or custom encoders, you may need to implement its redaction interface too, such as slog.LogValuer or a zap marshaler. The point is not that one method solves everything. The point is that the type should make accidental logging fail closed in the common paths.

Test the paths, not the intention

A useful test does not assert that String() returns [REDACTED]. That is too narrow. It fills every secret-bearing type with a canary value and tries to render it the ways developers actually render things while debugging.

Here is a stripped-down version:

package secrets_test

import (
    "encoding/json"
    "fmt"
    "strings"
    "testing"
)

type Secret string

func (Secret) String() string { return "[REDACTED]" }
func (Secret) GoString() string { return "[REDACTED]" }
func (Secret) MarshalJSON() ([]byte, error) {
    return []byte(`"[REDACTED]"`), nil
}

type DBConfig struct {
    Host     string `json:"host"`
    User     string `json:"user"`
    Password Secret `json:"password"`
}

func TestSecretDoesNotReachRenderedOutput(t *testing.T) {
    const canary = "CANARY_PASSWORD_DO_NOT_LOG_12345"

    cfg := DBConfig{
        Host:     "db.internal",
        User:     "app",
        Password: Secret(canary),
    }

    renderJSON := func(v any) string {
        b, err := json.Marshal(v)
        if err != nil {
            return err.Error()
        }
        return string(b)
    }

    cases := map[string]string{
        "fmt percent v":      fmt.Sprintf("%v", cfg),
        "fmt percent plus v": fmt.Sprintf("%+v", cfg),
        "fmt percent sharp v": fmt.Sprintf("%#v", cfg),
        "inside slice":       fmt.Sprintf("%#v", []DBConfig{cfg}),
        "inside map":         fmt.Sprintf("%#v", map[string]DBConfig{"primary": cfg}),
        "inside struct":      fmt.Sprintf("%#v", struct{ Config DBConfig }{cfg}),
        "wrapped error":      fmt.Errorf("connect failed: %v", cfg).Error(),
        "json":               renderJSON(cfg),
    }

    for name, got := range cases {
        if strings.Contains(got, canary) {
            t.Fatalf("%s leaked secret: %s", name, got)
        }
    }
}

This test catches boring mistakes. That is exactly why it is worth having.

If someone changes Password Secret back to Password string, the test fails. If someone adds a new secret field and forgets to wrap it, the test fails. If someone adds JSON output but forgets MarshalJSON, the test fails.

The failure is immediate and specific:

--- FAIL: TestSecretDoesNotReachRenderedOutput (0.00s)
    secrets_test.go:54: fmt percent sharp v leaked secret: secrets_test.DBConfig{Host:"db.internal", User:"app", Password:"CANARY_PASSWORD_DO_NOT_LOG_12345"}

That is much better than discovering the same string in CloudWatch, Datadog, or Elasticsearch after retention and replication have already done their work.

Know what this does not prove

This kind of test proves that your supported rendering paths do not expose the canary. It does not prove that no code anywhere can access the underlying bytes. Your application still needs to pass the cleartext to a database driver, API client, or vault SDK at some point.

It also does not cover every logger automatically. Structured loggers often avoid fmt and encode fields directly. For Go's log/slog, implement LogValue() on the secret type:

func (Secret) LogValue() slog.Value {
    return slog.StringValue("[REDACTED]")
}

Then add a test case that records through your actual logger and scans the emitted output for the canary. Do the same for zap, zerolog, or whatever your service uses.

Wire also treats related credential boundaries as testable behavior, for example failed connection attempts should not write partial credential state that later code might accidentally use.

There are other useful controls in the same family:

Zero plaintext buffers after use when you own the memory.
Do not persist failed credential configuration.
Check that credential references cannot point outside the scope they were granted.
Name tests after the security property, not the implementation detail.

A test named TestSecretDoesNotReachRenderedOutput explains the invariant you care about. A future refactor can change the implementation and still preserve the behavior.

Add a canary-based log-safety test around your own secret types, run it in CI, and extend it every time you add a new logger, serializer, or credential-bearing struct.

Use 1Password service accounts for automation, not shared logins

Anakin — Fri, 14 Aug 2026 10:00:10 +0000

Automating anything behind a login usually creates an awkward credential problem. Someone exports a password into an env var, a CI secret, or a database column. It works for a while, then the password rotates, the owner leaves, or the account gets locked because an old copy keeps retrying.

The better pattern is to avoid copying the credential at all. Give the automation its own scoped access to a password manager, store a reference to the item it needs, and read the secret only at the moment of use.

Use a service account, not a human account

A 1Password Service Account exists for machine access. It can access specific vaults, and you can revoke it without changing any person's account.

That matters more than it sounds. If you automate with a human login, you inherit everything messy about that user:

The automation can see everything the user can see.
Password changes can break jobs without warning.
Offboarding the user breaks the integration.
Audit logs point at the person, not the system doing the work.

A cleaner setup is a dedicated vault, for example Automation Logins, containing only the accounts your automation should use. Then create a Service Account with access to that vault and nothing else.

You can sanity check the scope with the 1Password CLI before wiring it into an app:

# Paste the token when prompted, rather than putting it in shell history
read -s OP_SERVICE_ACCOUNT_TOKEN
export OP_SERVICE_ACCOUNT_TOKEN

# Confirm the service account sees only the vaults you expect
op vault list --format json | jq -r '.[].name'

# Confirm a target item has standard Login fields
op item get 'Supplier Portal' \
  --vault 'Automation Logins' \
  --format json \
  | jq -r '.fields[] | select(.label == "username" or .label == "password") | .label'

If that command lists vaults you did not expect, fix the Service Account before doing anything else.

Wire uses this scoped Service Account pattern for login automation: the 1Password connection grants access to selected vaults, while each automated identity points at a specific login item.

Store references, not password copies

The important design choice is what your application stores.

Avoid this:

create table automated_identity (
  id uuid primary key,
  username text not null,
  password_ciphertext bytea not null
);

That schema creates a second source of truth. Once you copy the password, you own rotation, expiry, retry behavior, and cleanup. If the password changes in 1Password, your copy becomes stale. The failure often shows up later as repeated login failures, sometimes ending in an account lockout.

Prefer this:

create table credential_source (
  id uuid primary key,
  provider text not null,
  encrypted_service_account_token bytea not null,
  status text not null check (status in ('active', 'revoked'))
);

create table automated_identity (
  id uuid primary key,
  credential_source_id uuid not null references credential_source(id),
  vault_id text not null,
  item_id text not null
);

The identity stores a pointer: source, vault, item. It does not store the username or password. At login time, the app resolves the pointer, reads the current item from 1Password, uses the credential in memory, and discards it.

That one change makes rotation much less dramatic. Change the password in 1Password and the next login reads the new value. Existing web sessions may keep working until they expire because the target site already issued them, but the next sign-in does not need a database update or redeploy.

Know what the token grants

A 1Password ops_... token is not just a random API key. 1Password uses end-to-end encryption, so the token carries the material needed by the client SDK to authenticate and decrypt items the Service Account can access.

Treat it like a high-value secret:

Store it encrypted at rest.
Do not log it.
Do not put it in shell history.
Do not reuse it across environments unless you want shared blast radius.
Prefer separate Service Accounts for dev, prod, clients, or teams.

The token does not grant access to every vault in the 1Password account. It grants access to the vaults assigned to that Service Account. That is why scoping at creation time matters.

A useful implementation detail: verify the token before saving it. Make a live call to 1Password, list the accessible vaults, and only persist the encrypted token if that succeeds. Otherwise you end up with a connection record that looks configured but fails on first use.

Handle failure as state, not mystery

Credential integrations fail in predictable ways. Model those failures explicitly.

Common cases:

Error	Usually means	Fix
`SOURCE_TOKEN_REJECTED`	Token was revoked, rotated, expired, or mistyped	Re-enter or replace the Service Account token
`SOURCE_ITEM_NOT_FOUND`	Item was deleted, moved, or the stored item id is wrong	Rebind the identity to the correct item
`SOURCE_ITEM_MISSING_FIELDS`	The item is not a standard Login item, or lacks username/password fields	Convert or recreate it as a Login item
`SOURCE_REVOKED`	The Service Account no longer works	Mark dependent identities broken until reverified

The third one catches teams often. A Secure Note can contain text that looks like a username and password, but your integration may read only standard Login fields. If the item looks right in the UI but the SDK cannot find fields, check the item type.

Wire marks a 1Password source as revoked after a rejected read, so dependent identities fail visibly instead of retrying stale credentials in the background.

Revocation should not require vendor cooperation

The clean revocation path should start in the password manager. Delete or disable the Service Account in 1Password, and future reads should fail immediately.

Your app still needs to handle already-issued sessions. If the automation logged into a website yesterday, that site may keep accepting the session cookie until it expires. Revoking the Service Account prevents new password reads, but it does not automatically invalidate sessions held by third-party sites.

If you need immediate cutoff, do both:

Revoke the Service Account so no new credentials can be read.
Delete stored sessions in your automation system, or sign out all sessions on the target site.

The practical next step: create a dedicated 1Password vault with one non-critical test login, grant a Service Account only that vault, validate it with the CLI, then build your automation around item references rather than copied passwords.

Model Logins as Identities, Credentials, and Sessions

Anakin — Thu, 13 Aug 2026 10:00:01 +0000

"Store the user's login" sounds like one database row until the system has to handle real accounts. A customer has two accounts on the same supplier portal. A password rotates in 1Password. Someone imports a browser session because the password flow has MFA. A worker reuses cookies from a different IP and the site silently sends it back to login. If your model is just user_id, site, username, password, and cookies, you cannot answer the basic question: what account is this, what secret produced this session, and is it safe to refresh?

A cleaner model is to split the problem into three things: identity, credential, and session.

Separate the account from the login method

An identity should mean one account on one site, owned by one user or tenant. It is not the human user, and it is not the password.

That distinction matters when a user has multiple accounts on the same site:

create table identities (
  id uuid primary key,
  owner_id uuid not null,
  site text not null,
  label text not null,
  created_at timestamptz not null default now(),

  unique (owner_id, site, label)
);

create type credential_kind as enum ('CREDENTIALS', 'BROWSER_STATE');

create table credentials (
  id uuid primary key,
  identity_id uuid not null references identities(id) on delete cascade,
  kind credential_kind not null,
  vault_binding jsonb,
  encrypted_browser_state bytea,
  expires_at timestamptz,
  exit_ip inet,
  created_at timestamptz not null default now(),
  updated_at timestamptz not null default now(),

  unique (identity_id, kind)
);

The useful constraint is unique (identity_id, kind). One account can have a password-derived credential and an imported browser-state credential at the same time, but it cannot have two competing password credentials unless you model that explicitly.

That forces callers to choose what they want:

await runTask({
  identityId: "seller-us-prod",
  credentialKind: "CREDENTIALS"
});

await runTask({
  identityId: "seller-eu-prod",
  credentialKind: "BROWSER_STATE"
});

Wire uses this identity and credential split so tasks can choose the exact account and session type instead of relying on a hidden default.

Put the vault binding on the credential

The tempting shortcut is to attach the vault item to the identity:

{
  "identity": "supplier-account-123",
  "onePasswordItemId": "abc123"
}

That works until it doesn't. The account is stable. The way you authenticate to it is not.

You might start with a manually entered password, move the secret into 1Password, migrate to Azure Key Vault, or split username and password into separate entries because that is how your organisation stores them. None of those changes should modify the account object. They change how the next session gets minted.

A credential-level binding gives you that boundary:

{
  "identityId": "supplier-account-123",
  "kind": "CREDENTIALS",
  "vaultBinding": {
    "provider": "1password",
    "fields": {
      "username": {
        "itemId": "abc123",
        "field": "username"
      },
      "password": {
        "itemId": "abc123",
        "field": "password"
      }
    }
  }
}

It also lets you support messier vault layouts without changing the rest of the system:

{
  "provider": "aws-secrets-manager",
  "fields": {
    "username": {
      "secretId": "prod/supplier/username"
    },
    "password": {
      "secretId": "prod/supplier/password"
    }
  }
}

The binding should name the item, not a version. When the password rotates in the vault, the next login reads the current value. You do not need to update your application database just because the secret changed.

You should also reject accidental rebinding. If an existing credential points at one vault item and a request tries to attach a different one, return a conflict instead of quietly changing which secret backs a live account:

HTTP/1.1 409 Conflict
Content-Type: application/json

{
  "code": "IDENTITY_BINDING_MISMATCH",
  "message": "Credential is already bound to a different vault item"
}

That error saves you from a bad class of incident: the task still says "supplier production account", but the credential now logs into something else.

Treat sessions as derived state

A session is the thing that actually gets work done: cookies, tokens, local storage, and sometimes the proxy exit IP that created them.

That last field is easy to miss. Some sites bind sessions to network location. If you log in through one proxy exit and later send requests through another, the site may not return a clean 401. It might serve the login page with 200 OK, clear a cookie, or ask for MFA again. From the worker's point of view, the task just starts failing halfway through.

For password-derived sessions, store the exit IP that minted the session and reuse it:

type StoredSession = {
  encryptedState: Buffer;
  expiresAt?: Date;
  exitIp?: string;
};

async function loadSession(credentialId: string) {
  const credential = await db.credentials.find(credentialId);

  return {
    browserState: await decrypt(credential.encrypted_browser_state),
    proxyExit: credential.exit_ip ?? undefined
  };
}

Imported browser state is different. If a user uploads cookies from their local browser, you usually do not know the original exit IP. That session may pass every local test and then fail in production when a worker runs it from a data center IP. The model should make that difference visible rather than pretending all sessions have the same properties.

After each successful action, write updated cookies back to the credential. Many sites extend sessions on use, so this avoids unnecessary logins:

async function runWithStoredSession(credentialId: string, action: Action) {
  const session = await loadSession(credentialId);
  const browser = await launchBrowser({ proxyExit: session.proxyExit });

  await browser.context().addCookies(session.browserState.cookies);

  const result = await action(browser);

  if (await looksLoggedOut(browser)) {
    await markExpired(credentialId);
    return { ok: false, error: "AUTH_EXPIRED" };
  }

  const updatedState = await captureBrowserState(browser);
  await saveEncryptedState(credentialId, updatedState);

  return { ok: true, result };
}

Do not hide relogin behind a background timer unless you have a strong reason. On-demand login has better audit properties: a vault read happens because a task requested access, not because a scheduler decided to refresh every account at 3 AM.

Wire handles expired sessions this way: the task gets AUTH_EXPIRED, and a fresh login reads the vault only when something asks for that account again.

The tradeoff is more explicitness

This model adds rows, constraints, and decisions. Callers need to specify the identity and credential kind. Operators need to understand the difference between deleting a session, deleting an account, and disconnecting a vault.

For a single internal app with one login per user, this may be overbuilt. A normal OAuth table might be enough.

But if you automate third-party sites, support multiple accounts per site, or read secrets from a vault, the separation pays for itself. Keep accounts stable, attach secret bindings to credentials, and treat sessions as replaceable derived state.

A practical next step: sketch your current login table and mark which columns describe the account, which describe the secret source, and which describe the live session. If one row mixes all three, that is where your next auth migration will probably come from.

Design API Access Revocation So Users Know What Actually Gets Deleted

Anakin — Wed, 12 Aug 2026 10:00:01 +0000

Most APIs document the happy path for connecting a credential. Fewer document the exit path. That becomes a real problem during offboarding, incident response, or vendor reviews, because “disconnect” can mean anything from “stop refreshing this token” to “delete every stored secret right now.” Those are not the same operation, and treating them as one creates surprises.

If your system stores credentials, sessions, OAuth refresh tokens, service account bindings, or imported browser sessions, revocation needs a clear model. Users should know what stops working, what gets deleted, and what survives.

Separate the things people revoke

A common mistake is to expose one vague endpoint:

DELETE /connections/github

That endpoint tells the caller almost nothing. Does it delete the connection record? The tokens? The linked user identities? Cached data? Running jobs? If the answer depends on flags or internal state, someone will eventually use it wrong.

A clearer model splits revocation by scope:

DELETE /sessions/{session_id}
DELETE /identities/{identity_id}
DELETE /sources/{source_id}
DELETE /sources/{source_id}/identities

Those endpoints imply different outcomes:

Deleting a session removes one stored login or token. The account identity can remain. The next task should fail with something explicit, such as AUTH_EXPIRED, until someone reconnects.
Deleting an identity removes one connected account and every session underneath it. Other identities from the same source should remain untouched.
Disconnecting a source can remove the link to the vault or provider while keeping existing identities. That works for migration, but it is not complete offboarding.
Disconnecting a source and deleting its identities removes both the connection and all credentials sourced from it.

Wire uses this kind of split for credential revocation, where deleting a session, deleting an identity, and disconnecting a source are different operations with different failure behavior.

The important part is not the exact URL naming. The important part is that the destructive operation has its own shape.

Do not hide destructive behavior behind a flag

This is tempting:

DELETE /sources/{source_id}?delete_identities=true

It looks convenient, but it makes the safe path and the destructive path one parameter apart. That matters more than it seems. Query strings get copied into tickets. Bookmarks get reused. Retry code can accidentally preserve a flag that the next caller did not intend to send.

Prefer a separate endpoint for the irreversible operation:

DELETE /sources/{source_id}
DELETE /sources/{source_id}/identities

Then make the destructive endpoint atomic. If the source is removed but half the identities remain, you created a cleanup problem and probably a security exception.

In SQL, that usually means wrapping the operation in one transaction:

BEGIN;

DELETE FROM sessions
WHERE identity_id IN (
  SELECT id FROM identities WHERE source_id = $1
);

DELETE FROM identities
WHERE source_id = $1;

DELETE FROM sources
WHERE id = $1;

COMMIT;

If anything fails, rollback the whole thing:

ERROR: update or delete on table "identities" violates foreign key constraint

That error should not leave the system in a half-disconnected state. Either the credential set still exists, or it is gone.

Be precise about what “delete” means

“Deleted” often means “we set deleted_at and hide it in the UI.” That may be fine for many product records. It is not always fine for secrets.

For credential material, you should decide and document which of these you mean:

-- Soft delete
UPDATE sessions
SET deleted_at = now()
WHERE id = $1;

-- Hard delete
DELETE FROM sessions
WHERE id = $1;

A soft delete can support restore flows and audit workflows, but the encrypted token still exists. If an attacker later gets the encryption key and database, the material may still be recoverable.

A hard delete removes the row. You can still keep an audit event without keeping the secret:

{
  "event": "session.deleted",
  "session_id": "sess_123",
  "identity_id": "id_456",
  "deleted_by": "user_789",
  "deleted_at": "2026-08-05T10:15:00Z"
}

This gives compliance and debugging teams something to inspect without retaining the credential itself.

Also be clear about timing. If deletion happens synchronously, say so. If you enqueue it, expose the pending state:

{
  "status": "deletion_pending",
  "job_id": "job_abc"
}

Do not let users assume a nightly sweeper exists if it does not.

Treat external revocation as the final authority

If a customer grants access through their own vault, IAM role, OAuth app, or service account, the strongest revocation point is outside your system. They can delete the service account, remove the role assignment, revoke the OAuth app, or rotate the secret.

That matters because it does not depend on your API being reachable or your deletion path behaving correctly.

Your system should detect that external revocation and stop retrying forever. For example, if a read from a vault starts failing with 403 Forbidden, mark the source as revoked:

{
  "source_id": "src_123",
  "status": "revoked",
  "last_error": "VAULT_ACCESS_DENIED",
  "last_checked_at": "2026-08-05T10:20:00Z"
}

Then surface dependent identities as broken instead of returning stale data or hiding the failure behind a generic retry state. In systems like Wire, this distinction matters because vault revocation cuts off future reads even if stored identities still exist.

Running jobs are a separate edge case

Revocation usually affects the next operation, not necessarily the one already in progress.

If a worker loads a session at job start, deleting the session record 500 milliseconds later will not erase the copy already held in memory:

const session = await loadSession(sessionId);
await runLongScrape(session); // revocation during this call may not interrupt it

If that window matters, you need an active cancellation mechanism:

await jobs.cancelBySession(sessionId);

Even then, cancellation depends on workers checking cancellation state. For sessions controlled by a third-party site, the only reliable way to invalidate a copy already in use may be signing out everywhere or revoking the token at the original provider.

A practical checklist

For your own API, write down the revocation contract in plain terms:

What deletes one session?
What deletes an identity and all its sessions?
What disconnects a source but preserves identities?
What deletes both the source and sourced identities?
Which deletes are hard deletes, and which are soft deletes?
What happens to jobs already running?
What error does the next task return?
How does the system surface external revocation?

Then add tests for each path. The easiest test is also the most useful: create a source with two identities, add sessions under both, call each revocation endpoint, and assert exactly which rows remain.

Azure Key Vault RBAC: the role that breaks secret reads

Anakin — Tue, 11 Aug 2026 10:00:01 +0000

You create an app registration, give it access to Azure Key Vault, and your integration still fails with a 403 when it tries to read a secret. The confusing part is that some things work: the app can authenticate, the vault exists, and it may even list secret names. Then the actual get secret value call fails.

That usually means you granted the wrong Key Vault role.

Authentication is not vault access

Azure splits this into two separate checks.

Microsoft Entra ID answers: who is this app? If the client ID, tenant ID, or client secret is wrong, token acquisition fails. That is authentication.

Key Vault RBAC answers: what can this app read from this vault? A valid token can still get rejected by a specific vault. That is authorization.

The distinction matters when you handle errors:

Status	What it usually means	What to check
401	Entra rejected the principal	Tenant ID, client ID, client secret, expired credential
403	Principal exists but lacks access	Role assignment on this vault
404	Vault or secret was not found	Vault URL, secret name, deleted resource
429	Key Vault throttled the request	Retry policy and request volume

Do not collapse 401 and 403 into a generic “access denied” path. If you do, someone will rotate a client secret that was never broken, while the missing RBAC assignment stays missing.

Wire keeps this distinction when it reads Azure Key Vault secrets for account logins, so a missing vault role does not look like a rejected Entra credential.

The role name is the trap

Azure has a role called Key Vault Reader. It sounds like the right role for an app that needs to read from Key Vault.

It is not.

Key Vault Reader allows metadata access. An app can see information about keys, certificates, and secrets, but it cannot read secret values. This is why the failure feels inconsistent: listing may work, but fetching the value fails.

For secret values, use:

Key Vault Secrets User

Assign it per vault. Access to one vault does not imply access to another.

Here is the Azure CLI version I usually use because it makes the scope explicit:

APP_ID="00000000-0000-0000-0000-000000000000"
RESOURCE_GROUP="rg-prod"
VAULT_NAME="prod-login-vault"

SP_OBJECT_ID=$(az ad sp show \
  --id "$APP_ID" \
  --query id \
  -o tsv)

VAULT_ID=$(az keyvault show \
  --name "$VAULT_NAME" \
  --resource-group "$RESOURCE_GROUP" \
  --query id \
  -o tsv)

az role assignment create \
  --assignee-object-id "$SP_OBJECT_ID" \
  --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" \
  --scope "$VAULT_ID"

Then test the exact operation your app needs:

az keyvault secret show \
  --vault-name "$VAULT_NAME" \
  --name "github-login" \
  --query value \
  -o tsv

With the wrong role, you will see a failure like this:

(Forbidden) The user, group or application does not have secrets get permission on key vault

That message is the important part: secrets get permission. Listing secrets is not enough.

Treat the vault list as a security boundary

There is another practical edge: a service principal cannot ask the Key Vault data plane, “which vaults can I access?”

To enumerate vaults, you need Azure management-plane APIs and broader subscription permissions, often Reader at subscription or resource group scope. That is a much larger grant than “read these specific secrets.”

For most integrations, it is safer to configure the vault URLs explicitly and treat that list as the scope. Then validate every secret reference against it before making a Key Vault request.

A simple check looks like this:

const allowedVaultHosts = new Set([
  "prod-login-vault.vault.azure.net",
  "shared-automation-vault.vault.azure.net"
]);

export function assertAllowedVault(secretUrl: string) {
  const url = new URL(secretUrl);
  const host = url.hostname.toLowerCase();

  if (!allowedVaultHosts.has(host)) {
    throw new Error(`Vault is not in the configured allowlist: ${host}`);
  }

  return url;
}

Run this check on every resolution, not only when someone saves the configuration. Stored references can be edited, migrated, or generated by code. The read path should still enforce the boundary.

Because the configured vault URLs define scope, Wire rejects Key Vault references outside that list instead of trying to discover or follow them.

Store multi-field credentials deliberately

An Azure Key Vault secret is one opaque string. It is not a password-manager item with separate username and password fields.

If your app needs multiple fields, JSON is usually the least surprising option:

az keyvault secret set \
  --vault-name "prod-login-vault" \
  --name "github-login" \
  --value '{"username":"bot@example.com","password":"correct-horse-battery-staple"}'

Then parse it after reading the value:

type LoginSecret = {
  username: string;
  password: string;
};

function parseLoginSecret(value: string): LoginSecret {
  const parsed = JSON.parse(value);

  if (typeof parsed.username !== "string" || typeof parsed.password !== "string") {
    throw new Error("Secret must contain string username and password fields");
  }

  return parsed;
}

You can put a username in a tag if you need to, but do not put passwords in tags. Key Vault tags are metadata. People and services with metadata access can see them without being allowed to read secret values.

Let versionless references handle rotation

Key Vault keeps versions for each secret. If you fetch a secret without a version, Azure returns the latest version:

https://prod-login-vault.vault.azure.net/secrets/github-login

If you include a version, you pin the value:

https://prod-login-vault.vault.azure.net/secrets/github-login/635d6f...

For application logins and other runtime credentials, versionless references are usually what you want. Rotate the secret in Key Vault, and the next read gets the new value. Existing sessions keep whatever they already used until they expire.

Also turn on Key Vault diagnostic logs. Once logs flow into Log Analytics, you can query reads from your own tenant:

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.KEYVAULT"
| where OperationName in ("SecretGet", "SecretList")
| project TimeGenerated, Resource, OperationName, identity_claim_appid_g, ResultType
| order by TimeGenerated desc

Before you ship an integration, test three cases on purpose: wrong client secret, missing Key Vault Secrets User on one vault, and a valid read from each configured vault. If those produce different errors in your app, your future incident response will be much less annoying.

Authenticated scraping should use sessions, not passwords

Anakin — Fri, 07 Aug 2026 10:00:01 +0000

Ask an agent to fetch your order history and you’ll often get a summary of the public storefront instead. It can read product pages, shipping policies, and FAQs, then it hits the login wall and stops. The useful data is usually behind authentication: invoices, order status, supplier pricing, account-specific quotes. If your automation cannot authenticate safely, it is reading the brochure.

The password is the wrong object to hand over

A password gives too much authority.

It can often change account settings, update recovery emails, add payment methods, or lock the owner out. It may also be reused across sites, which turns one bad integration into a broader incident. Even if your code never intentionally logs it, passwords tend to pass through dangerous places: request bodies, debug logs, retry queues, traces, prompt context, and vendor dashboards.

Driving a real browser does not fix that. Chrome still needs the password from somewhere. You have only added a rendering engine, a higher bill, and more ways for CSS selectors to break.

The better target is the session.

When you log in, the site checks the password once and returns cookies or tokens that prove the login already happened. Every later request sends the session, not the password. That distinction matters because a session is narrower than a password:

It usually works for one site only.
It expires after hours or days.
The user can revoke it by signing out everywhere.
It often cannot perform high-risk actions without a password re-prompt.

That last point is not universal. Some portals let a session change email addresses or payout details. You need to test the specific site. But in most scraping and workflow automation cases, a session is still the least-bad thing to store.

A safer shape for authenticated scraping

Separate login from work execution. The login component touches the secret. The worker never does.

A minimal version looks like this:

// login-service.ts
import { chromium } from "playwright";
import { encrypt } from "./crypto";
import { saveSession } from "./session-store";
import { readSecret } from "./vault";

export async function createSession(identityId: string) {
  const { username, password } = await readSecret(identityId); // memory only

  const browser = await chromium.launch();
  const page = await browser.newPage();

  await page.goto("https://supplier.example.com/login");
  await page.fill("input[name=email]", username);
  await page.fill("input[name=password]", password);
  await page.click("button[type=submit]");

  await page.waitForURL("**/account", { timeout: 15000 });

  const storageState = await page.context().storageState();
  await browser.close();

  const encrypted = await encrypt(JSON.stringify(storageState));
  const sessionId = await saveSession(identityId, encrypted);

  return { sessionId };
}

Then the worker receives only the stored browser state:

// worker.ts
import { chromium } from "playwright";
import { decrypt } from "./crypto";
import { loadSession } from "./session-store";

export async function fetchOrders(sessionId: string) {
  const encrypted = await loadSession(sessionId);
  const storageState = JSON.parse(await decrypt(encrypted));

  const browser = await chromium.launch();
  const context = await browser.newContext({ storageState });
  const page = await context.newPage();

  await page.goto("https://supplier.example.com/orders");

  if (page.url().includes("/login")) {
    throw new Error("AUTH_EXPIRED");
  }

  const rows = await page.locator("[data-order-row]").allTextContents();

  const refreshedState = await context.storageState();
  // Persist this if the site rotated or refreshed cookies during the request.

  await browser.close();
  return rows;
}

The important part is not Playwright. Use HTTP clients if the site supports it. The important part is the boundary: one component can read the credential, mint a session, encrypt it, and discard the credential. The scraping worker has no code path to the password.

Wire follows this same session-first model for authenticated scraping: login uses the credential once, while task workers operate on cookies and tokens rather than passwords.

Keep the password in a vault if you can

Even “we only use the password once” still has a handoff problem. Someone typed the secret into your system, sent it over your API, or stored it in your database briefly enough that you hope it did not leak.

For teams with credential policies, that is often unacceptable. The better version is to store a reference to a vault item, not the secret itself.

For example, your database stores this:

{
  "identityId": "supplier-portal-prod",
  "vaultProvider": "1password",
  "vaultName": "Finance Shared",
  "itemName": "Acme Supplier Portal",
  "usernameField": "username",
  "passwordField": "password"
}

At login time, the login service asks the vault for the current value, uses it in memory, then drops it. Rotation becomes boring. If IT changes the password in the vault, the next login reads the new value. You do not update your scraper configuration.

This also improves auditability. Vault providers already log secret reads. If you need to know when the automation accessed a credential, you inspect the vault audit trail rather than trusting an application log generated by the same system you are auditing.

Sessions still need real failure handling

Authenticated scraping fails in predictable ways. Treating every failure as login failed makes the caller useless.

Return specific errors:

type AuthFailureCode =
  | "AUTH_EXPIRED"
  | "MFA_REQUIRED"
  | "CAPTCHA_REQUIRED"
  | "ACCOUNT_LOCKED"
  | "LOGIN_PAGE_CHANGED";

Each code implies a different action:

AUTH_EXPIRED: create a new session, possibly by reading the vault again.
MFA_REQUIRED: ask a human, or mark the site unsupported for unattended login.
CAPTCHA_REQUIRED: stop retrying. More retries often make this worse.
ACCOUNT_LOCKED: tell the account owner to fix it at the source site.
LOGIN_PAGE_CHANGED: update the integration because the site changed its flow.

Wire returns explicit auth failure codes such as AUTH_EXPIRED, MFA_REQUIRED, and LOGIN_PAGE_CHANGED, which is the difference between a retryable session problem and a site integration problem.

What to build next

If you are adding authenticated scraping to an agent or workflow system, start by drawing the credential boundary. The worker should accept a sessionId, not a password. Store encrypted session state, refresh it when the site does, and return explicit auth errors when it stops working.

Then move the original credential into a vault and store only a reference to it. That one change removes the worst object from your application database.

1Password vs Azure Key Vault: The Real Difference Is Scope

Anakin — Thu, 06 Aug 2026 10:00:01 +0000

A secret-backed automation often fails in a way that looks simple: login stopped working. The hard part is figuring out whether the credential is dead, the app lost access to one vault, the secret moved, or the secret is shaped differently than the code expects.

If you are choosing between 1Password and Azure Key Vault for automated logins, the interesting difference is not that both can store encrypted secrets. They can. The difference is how each system answers: what is this integration allowed to read?

1Password puts scope in the token

With 1Password, you create a Service Account, grant it access to specific vaults, and use the generated token in your automation.

That token is self-scoping. The service account token already carries the information needed to know which vaults it can reach. If you remove a vault from the service account, reads from that vault stop. If you delete the service account, all reads using that token fail.

That makes setup simple, but it also means the important access decision happens inside 1Password. Your automation usually does not need its own allowlist of vault IDs because the token already defines the boundary.

This model fits teams where the people who manage shared credentials are also the people wiring up automation. It is quick to reason about: create a service account, grant vaults, store the token, rotate or revoke when needed.

The tradeoff is governance location. If your organization expects access review, role assignment, and audit trails to live in Azure, 1Password may be outside the system auditors already inspect.

Azure makes you define the scope explicitly

Azure Key Vault works differently. An app registration in Microsoft Entra ID proves who the automation is, but each Key Vault still decides what that identity can read.

There is another important wrinkle: Azure does not provide a simple data-plane API that says, “list every vault this principal can access.” You can enumerate vaults through Azure’s management plane, but that requires broader subscription-level permissions. For a secrets integration, that is often more access than you wanted to grant.

So the safer pattern is to provide the vault URLs yourself and treat that list as the integration scope. Wire uses that pattern for Azure Key Vault connections: the configured vault URLs act as the allowed set, even if the Azure credential could technically access more.

A minimal Azure setup looks like this:

# Values you already have after creating an Entra app registration
CLIENT_ID="00000000-0000-0000-0000-000000000000"
TENANT_ID="11111111-1111-1111-1111-111111111111"
VAULT_NAME="my-login-vault"
SUBSCRIPTION_ID="22222222-2222-2222-2222-222222222222"

VAULT_SCOPE="/subscriptions/$SUBSCRIPTION_ID/resourceGroups/my-rg/providers/Microsoft.KeyVault/vaults/$VAULT_NAME"

az role assignment create \
  --assignee "$CLIENT_ID" \
  --role "Key Vault Secrets User" \
  --scope "$VAULT_SCOPE"

The role matters. Key Vault Reader sounds right, but it only grants metadata access. With that role, your app may list secret names and then get a 403 Forbidden when it tries to read the value.

For automated logins, that failure is easy to misdiagnose. The client secret is valid. The tenant is correct. The vault exists. The missing piece is the data-plane role that allows reading secret values.

401 and 403 mean different things

In Azure, authentication and authorization are separate.

Entra ID authenticates the app and issues a token. Key Vault authorizes that token for specific actions on a specific vault. A credential can be valid and still fail on one vault.

That means your error handling should not collapse everything into “access denied.” Treat these cases differently:

Status	Likely cause	What to do
401	Entra rejected the app credential	Check tenant ID, client ID, client secret expiry, or disabled app
403	The app authenticated but lacks vault permission	Check RBAC on that vault, usually `Key Vault Secrets User`
404	Vault, secret, or version was not found	Check URL, secret name, and version references
429	Key Vault throttled the request	Respect `Retry-After` and retry later

This distinction saves time. If one vault returns 403 but another works, regenerating the app’s client secret will not help. The app is authenticated. It just lacks a role assignment on that vault.

Secret shape affects your binding code

1Password has structured Login items. A login can have a username, password, URL, and custom fields. Automation can ask for the fields it needs.

Azure Key Vault secrets are opaque strings plus metadata tags. The value might be a plain password, a JSON object, or one part of a larger credential split across several secrets.

For example:

az keyvault secret set \
  --vault-name my-login-vault \
  --name github-login \
  --value '{"username":"bot@example.com","password":"s3cr3t"}'

az keyvault secret set \
  --vault-name my-login-vault \
  --name github-password \
  --value 's3cr3t' \
  --tags username='bot@example.com'

Both can work, but they imply different parsing rules. If the value is JSON, your code can map keys to fields. If the value is a bare string, you need to decide whether the string is the password, token, or something else.

One caution: Key Vault tags are metadata, not secrets. A username in a tag is usually fine. A password in a tag is not. Anyone who can list metadata can read tags.

Wire handles Azure secret values by exposing parsed JSON keys when possible, falling back to a value field for plain strings, and namespacing tags so metadata cannot shadow real secret fields.

Which one should you use?

Choose 1Password when your team already manages application credentials there and you want the smallest operational surface. The setup is shorter because the service account token carries its own vault scope. Revocation is also straightforward: remove the service account or change its vault grants.

Choose Azure Key Vault when Azure is already your control plane. It fits better when access should be granted through Entra ID and Azure RBAC, when you need per-vault role assignments, when secret versioning matters, or when you need Azure sovereign cloud support.

Azure’s extra setup is not accidental complexity. It comes from separating identity, vault authorization, and vault discovery. That separation gives you more governance hooks, but it also gives you more ways to fail.

One practical next step: write down the exact scope model before wiring anything up. For each automation, list the vaults it may read, the role or service account that grants access, the expected secret shape, and the error you expect when access is revoked. Then test the revoke path, not just the successful login.

Browser automation vs network replay for portal workflows

Anakin — Tue, 04 Aug 2026 10:00:01 +0000

If you have ever automated a third-party portal, you have probably hit the same wall: the thing works in staging, runs fine for a few days, then fails because a button moved, a modal appeared, a session expired, or the site shipped a minor frontend change that broke your selector. The hard part is not clicking the button once. The hard part is making the workflow boring enough to run hundreds or thousands of times.

There are three common layers where people run this kind of automation: browser agents, deterministic browser scripts, and network replay. They can all work. They fail in different ways.

Browser agents are the fastest way to get a first result

A browser agent drives the UI the way a person would. It reads the page, reasons about the next action, clicks, types, waits, and adapts when the layout changes.

That makes it useful when you know almost nothing about the target system. If an internal ops team says, "Log into this payer portal, find this patient, download the eligibility result," an agent can often attempt that without a developer reverse-engineering the site first.

The cost is predictability. Every page transition can involve model calls, screenshots, DOM extraction, or both. Latency tends to be measured in seconds or minutes, not milliseconds. Failures are also harder to classify because the runtime is making judgment calls.

A typical failure looks like this:

Agent stopped: unable to determine next action
Last observation: "Your session is about to expire"
Current goal: "Submit enrollment form"

A human would click "Continue" and move on. The agent may or may not, depending on whether it recognizes the interruption and whether its instructions allow it.

This layer is still the right starting point for unknown portals, desktop apps, Citrix sessions, and workflows where no stable web API exists. Just do not confuse "it completed once" with "it is production-ready."

Deterministic browser scripts reduce cost, but inherit UI fragility

Once you understand a workflow, you can replace agent reasoning with a script. That usually means Playwright, Puppeteer, Selenium, or a vendor-generated equivalent.

import { chromium } from "playwright";

const browser = await chromium.launch();
const page = await browser.newPage();

await page.goto("https://example-portal.com/login");
await page.fill('input[name="username"]', process.env.PORTAL_USER!);
await page.fill('input[name="password"]', process.env.PORTAL_PASS!);
await page.click('button[type="submit"]');

await page.waitForURL("**/dashboard");
await page.fill('input[aria-label="Patient ID"]', "123456");
await page.click('text=Search');
await page.click('text=Download PDF');

await browser.close();

This is cheaper and easier to debug than a full agent loop. You can log every step, retry specific operations, and write tests around known states.

But it still depends on the rendered UI. If the portal changes input[aria-label="Patient ID"] to input[placeholder="Member ID"], Playwright will fail with something like:

TimeoutError: locator.fill: Timeout 30000ms exceeded.
Call log:
  - waiting for locator('input[aria-label="Patient ID"]')

You can make scripts less brittle with better locators, explicit waits, screenshots on failure, and fallback selectors. You cannot remove the basic dependency: the UI is now part of your integration contract, even though the site owner never promised to keep it stable.

Network replay is usually the production layer

Most web portals are frontends over HTTP APIs. When you click "Search," the browser often sends a JSON request to a private endpoint. If you can identify that request and reproduce it safely, you can skip the UI entirely.

In Chrome DevTools, this usually starts in the Network tab:

POST https://example-portal.com/api/member/search
content-type: application/json
cookie: session=...

{ "memberId": "123456" }

Then you test whether the call works outside the browser:

curl 'https://example-portal.com/api/member/search' \
  -H 'content-type: application/json' \
  -H "cookie: session=$PORTAL_SESSION" \
  -d '{"memberId":"123456"}'

If that returns the same data, you have a much better automation path. No selectors. No screenshots. No waiting for animations. No LLM deciding whether a modal matters.

The tradeoff is setup work. You need to understand auth, CSRF tokens, pagination, rate limits, request signing, and any backend validation the frontend hides. Some portals make this easy. Others rotate tokens, bind requests to browser fingerprints, or put meaningful state in opaque headers.

A network-layer system such as Wire packages that discovery behind catalog actions, so callers invoke an action_id instead of maintaining private endpoint details themselves.

The API shape often looks like an async job because the provider handles login, retries, and target-site latency behind the scenes:

curl -X POST https://api.openwire.sh/v1/wire/task \
  -H "X-API-Key: $ANAKIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "action_id": "amazon.search_products",
    "params": { "query": "wireless keyboard" }
  }'

{
  "status": "processing",
  "job_id": "3f8aa45d-6ea3-4107-88ce-7f39ecf48a84",
  "poll_url": "/v1/wire/jobs/3f8aa45d-6ea3-4107-88ce-7f39ecf48a84"
}

That pattern matters more than the specific vendor. You submit a semantic action, poll by job ID, and receive structured data. Your code depends on a schema, not on a button label.

The schema is part of the product, even for internal tools

Developers often focus on getting the data once, then regret not defining the output contract. If one run returns price, another returns amount, and a third returns pricing.current, your downstream pipeline now has to guess.

For production workflows, treat the response shape like any other API contract:

{
  "products": [
    {
      "title": "Logitech MX Keys",
      "price": 99.99,
      "currency": "USD",
      "rating": 4.7
    }
  ]
}

Validate it at the boundary:

import { z } from "zod";

const Product = z.object({
  title: z.string(),
  price: z.number(),
  currency: z.string(),
  rating: z.number().nullable()
});

const SearchResult = z.object({
  products: z.array(Product)
});

const parsed = SearchResult.parse(apiResponse.data);

This catches drift early. Without validation, the first symptom may be a bad report, a failed billing job, or an empty CSV sent to a customer.

Catalog-based tools like Wire are strongest when the action has a documented input and output schema that stays stable across callers.

How to choose the layer

Use a browser agent when you need a first result on an unknown system, especially if the workflow spans desktop software, virtualized apps, or messy human-oriented screens.

Use a deterministic browser script when the workflow is known, volume is moderate, and UI changes are acceptable operational risk.

Use network replay when the workflow is high-volume, latency-sensitive, or feeding another system that expects consistent structured data.

The practical next step: open DevTools on one workflow you currently automate through the browser, perform the action manually, and inspect the Network tab. If the real operation is a small JSON request, move that path out of the UI before you spend more time making selectors prettier.

Compare Scraping APIs by Cost per Delivered Record, Not Request Price

Anakin — Thu, 30 Jul 2026 10:00:00 +0000

A scraping API that costs $0.50 per 1,000 requests can still be more expensive than one that advertises a higher price. The difference shows up when targets block you, return CAPTCHA pages, require browser rendering, or force retries that still consume credits.

Price per request is the wrong unit

For high-volume scraping, the useful metric is cost per delivered record.

A request is not the same thing as a usable result. A request can fail with a 403, time out, return a login wall, or return a CAPTCHA page with HTTP 200. If your pipeline treats all 200 responses as success, you may pay for junk and store junk.

A better unit is:

cost_per_delivered_1k = total_spend / successful_records * 1000

That metric includes the parts pricing pages usually hide:

Failed requests that still consume credits
Retries that count as additional requests
JavaScript rendering multipliers
Residential proxy premiums
Parsing failures after the fetch succeeds

If vendor A charges $0.50 per 1,000 requests and vendor B charges $0.70, vendor A only wins if both deliver records at roughly the same success rate and both bill failures the same way.

Model failures before comparing vendors

Say you submit 1,000,000 URLs at $0.50 per 1,000 requests.

At a 10% failure rate, 900,000 URLs return usable data and 100,000 fail. With no refund policy, you still pay for all 1,000,000 attempts:

1,000,000 / 1,000 * $0.50 = $500
$500 / 900,000 * 1,000 = $0.56 per delivered 1K

With pay-per-success or automatic failure refunds, you pay only for the 900,000 successful records:

900,000 / 1,000 * $0.50 = $450
$450 / 900,000 * 1,000 = $0.50 per delivered 1K

That is only $50 per million requests at a 10% failure rate. At 10 million requests per month, it becomes $500. At 20% failures, which is common on job boards, marketplaces, and bot-protected product pages, the difference gets large enough to change the vendor decision.

Wire fits this specific failure-cost problem by running extraction as jobs and refunding failed runs, so blocked or timed-out jobs do not become paid records.

Here is a small calculator I use when comparing providers:

def delivered_cost(
    requests: int,
    price_per_1k: float,
    failure_rate: float,
    refund_failures: bool,
    js_multiplier: float = 1.0,
    retry_multiplier: float = 1.0,
):
    successful = requests * (1 - failure_rate)

    if refund_failures:
        billable = successful
    else:
        billable = requests * retry_multiplier

    spend = (billable / 1000) * price_per_1k * js_multiplier
    return {
        "successful_records": int(successful),
        "spend": round(spend, 2),
        "cost_per_delivered_1k": round(spend / successful * 1000, 4),
    }

print(delivered_cost(
    requests=1_000_000,
    price_per_1k=0.50,
    failure_rate=0.10,
    refund_failures=False,
))

print(delivered_cost(
    requests=1_000_000,
    price_per_1k=0.50,
    failure_rate=0.10,
    refund_failures=True,
))

This is not a benchmark. It is a sanity check. You still need to measure your own target set.

JavaScript rendering changes the math

Browser rendering is the other place costs jump.

Many APIs charge a multiplier when you enable headless Chrome or Playwright. A 5x multiplier turns $0.50 per 1,000 requests into $2.50 per 1,000 requests. Some providers use higher multipliers for premium proxies, browser rendering, or both together.

The mistake is enabling browser rendering globally because a few targets need it.

Split your targets into modes:

{
  "static_http": [
    "https://example.com/products/1",
    "https://example.com/products/2"
  ],
  "needs_browser": [
    "https://example.com/search?q=laptop"
  ]
}

Then test both paths. If the price is present in the initial HTML, use HTTP. If the response contains only an app shell like this, you probably need rendering:

<div id="root"></div>
<script src="/assets/app.bundle.js"></script>

Even then, check whether the frontend calls a JSON endpoint you can request directly. Browser rendering should be the fallback, not the default.

Session-based browser billing can be cheaper for workflows that reuse cookies, perform login, paginate, or extract several pages in one browser session. It can be worse for short one-page jobs if the provider rounds session time up to a minimum billing interval. Measure both patterns.

Test with a small target set

Before signing an annual contract or migrating a pipeline, run a representative sample. I usually want at least a few hundred URLs per target type:

Static product pages
Search result pages
Geo-specific pages
Authenticated pages
Known bot-protected pages

Log outcomes separately. Do not only log HTTP status.

import requests

BAD_MARKERS = [
    "captcha",
    "access denied",
    "verify you are human",
    "unusual traffic",
]

def classify_response(resp):
    text = resp.text.lower()

    if resp.status_code in (401, 403, 429):
        return "blocked"

    if any(marker in text for marker in BAD_MARKERS):
        return "blocked_content"

    if resp.status_code >= 500:
        return "server_error"

    if len(resp.text) < 500:
        return "suspiciously_small"

    return "usable"

resp = requests.get("https://example.com", timeout=30)
print(resp.status_code, classify_response(resp))

This catches the annoying case where the provider returns HTTP 200, but the page content is a CAPTCHA or bot challenge. If the API bills that as success, your real cost is higher than the invoice suggests because your parser still has no usable data.

For teams that need reliable extraction rather than raw HTML fetches, Wire is relevant because its job model makes failure handling, polling, and result accounting explicit instead of hiding them behind a single request count.

What to ask before committing

Ask vendors specific billing questions. Vague answers are a signal to keep testing.

Does a 403 consume credits?
Does a timeout consume credits?
Does a CAPTCHA page returned as HTTP 200 count as success?
Are internal retries billed separately?
What is the multiplier for JavaScript rendering?
Are residential proxies included or billed at a premium?
Can I export per-request status and credit usage?

The cheapest scraping API is usually not the one with the lowest advertised request price. It is the one with the lowest cost per usable record for your target mix.

Take a sample of your real URLs, run them through two or three providers, classify the responses yourself, and calculate cost per delivered 1,000 records before you compare pricing pages.

Session Reuse and Browser Mode Decisions for Web Data Pipelines

Anakin — Wed, 29 Jul 2026 10:00:01 +0000

A scraper that works on public HTML often fails the first time you point it at real customer data. Instead of the dashboard content, you index a login page. Instead of a product catalog, you get region-specific defaults. Instead of JSON, your parser throws because the page is just a JavaScript shell.

These are not parsing problems first. They are state and execution problems.

Know what kind of page you are fetching

Before reaching for a browser, classify the failure.

Common symptoms:

Symptom	What it usually means
`401 Unauthorized`	Missing or expired credentials
HTML contains `Sign in` or `Log in`	You fetched the login state, not the authenticated state
`403 Forbidden`	Blocked client, wrong region, or missing auth
Empty `<div id="root"></div>` page	Content renders client-side after hydration
Different price than expected	Geo-routing or account tier changed the response
`Unexpected token < in JSON at position 0`	Code expected JSON but received HTML, often an error page

Do this classification before adding retries. Retrying a login page gives you the same login page.

Reuse sessions only inside clear boundaries

Authenticated scraping needs session state: cookies, local storage, sometimes an assigned exit region. Re-authenticating for every fetch wastes time and can trigger security systems. Reusing sessions too broadly creates a worse problem: data leaks across users or tenants.

A reasonable policy is:

Same domain, same user, same auth context: reuse the session.
Same domain, different role or account: create a new session.
Different customer or tenant: always create a new session.
Logged-in and logged-out views: treat them as separate sessions.
Expired or revoked credentials: delete the session and require re-authentication.

You can encode that policy directly instead of leaving it to caller discipline:

type SessionKeyInput = {
  tenantId: string;
  userId: string;
  domain: string;
  authContext: 'anonymous' | 'customer' | 'admin';
  country?: string;
};

function sessionKey(input: SessionKeyInput) {
  return [
    input.tenantId,
    input.userId,
    input.domain,
    input.authContext,
    input.country ?? 'default'
  ].join(':');
}

function shouldReuseSession(a: SessionKeyInput, b: SessionKeyInput) {
  return sessionKey(a) === sessionKey(b);
}

The key point is tenant isolation. If two customers use the same upstream SaaS app, they still need separate sessions. Shared cookies are a security bug, not an optimization.

Set an expiration policy too. Ninety days may be fine for some internal tools. Banking or healthcare workflows may need much shorter retention. Whatever you choose, make deletion explicit and irreversible from the application point of view.

Pick standard HTTP before browser automation

Browser mode costs more because it does more. It starts a browser context, runs JavaScript, waits for the page to settle, and sometimes interacts with the page. That is useful when the content only exists after client-side execution. It is wasteful when the server already returns usable HTML.

A good default decision tree looks like this:

type FetchMode = 'http' | 'browser';

type PageSignals = {
  status: number;
  html: string;
  needsScroll?: boolean;
  hasCaptcha?: boolean;
};

function chooseFetchMode(signals: PageSignals): FetchMode {
  if (signals.status === 403) return 'browser';
  if (signals.hasCaptcha) return 'browser';
  if (signals.needsScroll) return 'browser';

  const looksLikeJsShell =
    signals.html.includes('<div id="root"></div>') ||
    signals.html.includes('<div id="app"></div>');

  if (looksLikeJsShell && signals.html.length < 5000) {
    return 'browser';
  }

  return 'http';
}

This is intentionally conservative. A React app can still server-render useful HTML, and a short HTML document is not always a failure. Log the decision and sample responses so you can tune the heuristics per domain.

If your extraction provider can authenticate and extract at the network layer, Wire belongs on the non-browser side of this decision: use browser automation only when page behavior requires a real browser.

Keep geo-routing part of the session identity

Region-specific content creates subtle bugs. The request succeeds, the parser works, and the model still answers incorrectly because it saw the US catalog instead of the India catalog.

Treat country or region as part of the request and session identity:

{
  "url": "https://example.com/pricing",
  "session_id": "tenant_42:user_7:example.com:customer:in",
  "country": "in"
}

Do not reuse a country=us authenticated session for a country=in fetch unless you have verified the upstream application ignores region. Many sites bind catalog, currency, legal text, and inventory to both account and IP region.

Cache by everything that changes the response

Caching only by URL is usually wrong for authenticated or regional pages.

Use a cache key that includes the meaningful request dimensions:

import crypto from 'node:crypto';

function cacheKey(input: {
  url: string;
  tenantId: string;
  authContext: string;
  country?: string;
  headers?: Record<string, string>;
}) {
  const material = JSON.stringify({
    url: input.url,
    tenantId: input.tenantId,
    authContext: input.authContext,
    country: input.country ?? null,
    acceptLanguage: input.headers?.['accept-language'] ?? null
  });

  return crypto.createHash('sha256').update(material).digest('hex');
}

For public pages, a URL-only cache may be enough. For logged-in dashboards, pricing pages, or catalogs, include tenant, auth context, country, and any header that changes the response.

Monitor the failures that indicate wrong mode

Track more than success rate. A request that returns 200 OK with a login page is still a failed extraction.

Useful production checks include:

Percentage of fetched pages that contain login markers.
Percentage of pages with very low text length.
Browser fallback rate per domain.
Session expiration rate.
Region mismatch reports from downstream validation.

If browser fallback climbs from 5% to 60% for a domain, something changed. Maybe the site moved rendering client-side. Maybe it started blocking your HTTP client. Either way, you want to notice before stale or empty documents enter the index.

The practical next step is to add response classification before parsing: detect login pages, JS shells, region mismatches, and empty content, then choose session reuse or browser execution based on those signals.

The full breakdown is here if you want the complete picture.