Authenticated scraping should use sessions, not passwords

Ask an agent to fetch your order history and you’ll often get a summary of the public storefront instead. It can read product pages, shipping policies, and FAQs, then it hits the login wall and stops. The useful data is usually behind authentication: invoices, order status, supplier pricing, account-specific quotes. If your automation cannot authenticate safely, it is reading the brochure.

The password is the wrong object to hand over

A password gives too much authority.

It can often change account settings, update recovery emails, add payment methods, or lock the owner out. It may also be reused across sites, which turns one bad integration into a broader incident. Even if your code never intentionally logs it, passwords tend to pass through dangerous places: request bodies, debug logs, retry queues, traces, prompt context, and vendor dashboards.

Driving a real browser does not fix that. Chrome still needs the password from somewhere. You have only added a rendering engine, a higher bill, and more ways for CSS selectors to break.

The better target is the session.

When you log in, the site checks the password once and returns cookies or tokens that prove the login already happened. Every later request sends the session, not the password. That distinction matters because a session is narrower than a password:

• It usually works for one site only.
• It expires after hours or days.
• The user can revoke it by signing out everywhere.
• It often cannot perform high-risk actions without a password re-prompt.

That last point is not universal. Some portals let a session change email addresses or payout details. You need to test the specific site. But in most scraping and workflow automation cases, a session is still the least-bad thing to store.

A safer shape for authenticated scraping

Separate login from work execution. The login component touches the secret. The worker never does.

A minimal version looks like this:

// login-service.ts
import { chromium } from "playwright";
import { encrypt } from "./crypto";
import { saveSession } from "./session-store";
import { readSecret } from "./vault";

export async function createSession(identityId: string) {
  const { username, password } = await readSecret(identityId); // memory only

  const browser = await chromium.launch();
  const page = await browser.newPage();

  await page.goto("https://supplier.example.com/login");
  await page.fill("input[name=email]", username);
  await page.fill("input[name=password]", password);
  await page.click("button[type=submit]");

  await page.waitForURL("**/account", { timeout: 15000 });

  const storageState = await page.context().storageState();
  await browser.close();

  const encrypted = await encrypt(JSON.stringify(storageState));
  const sessionId = await saveSession(identityId, encrypted);

  return { sessionId };
}

Then the worker receives only the stored browser state:

// worker.ts
import { chromium } from "playwright";
import { decrypt } from "./crypto";
import { loadSession } from "./session-store";

export async function fetchOrders(sessionId: string) {
  const encrypted = await loadSession(sessionId);
  const storageState = JSON.parse(await decrypt(encrypted));

  const browser = await chromium.launch();
  const context = await browser.newContext({ storageState });
  const page = await context.newPage();

  await page.goto("https://supplier.example.com/orders");

  if (page.url().includes("/login")) {
    throw new Error("AUTH_EXPIRED");
  }

  const rows = await page.locator("[data-order-row]").allTextContents();

  const refreshedState = await context.storageState();
  // Persist this if the site rotated or refreshed cookies during the request.

  await browser.close();
  return rows;
}

The important part is not Playwright. Use HTTP clients if the site supports it. The important part is the boundary: one component can read the credential, mint a session, encrypt it, and discard the credential. The scraping worker has no code path to the password.

Wire follows this same session-first model for authenticated scraping: login uses the credential once, while task workers operate on cookies and tokens rather than passwords.

Keep the password in a vault if you can

Even “we only use the password once” still has a handoff problem. Someone typed the secret into your system, sent it over your API, or stored it in your database briefly enough that you hope it did not leak.

For teams with credential policies, that is often unacceptable. The better version is to store a reference to a vault item, not the secret itself.

For example, your database stores this:

{
  "identityId": "supplier-portal-prod",
  "vaultProvider": "1password",
  "vaultName": "Finance Shared",
  "itemName": "Acme Supplier Portal",
  "usernameField": "username",
  "passwordField": "password"
}

At login time, the login service asks the vault for the current value, uses it in memory, then drops it. Rotation becomes boring. If IT changes the password in the vault, the next login reads the new value. You do not update your scraper configuration.

This also improves auditability. Vault providers already log secret reads. If you need to know when the automation accessed a credential, you inspect the vault audit trail rather than trusting an application log generated by the same system you are auditing.

Sessions still need real failure handling

Authenticated scraping fails in predictable ways. Treating every failure as login failed makes the caller useless.

Return specific errors:

type AuthFailureCode =
  | "AUTH_EXPIRED"
  | "MFA_REQUIRED"
  | "CAPTCHA_REQUIRED"
  | "ACCOUNT_LOCKED"
  | "LOGIN_PAGE_CHANGED";

Each code implies a different action:

• AUTH_EXPIRED: create a new session, possibly by reading the vault again.
• MFA_REQUIRED: ask a human, or mark the site unsupported for unattended login.
• CAPTCHA_REQUIRED: stop retrying. More retries often make this worse.
• ACCOUNT_LOCKED: tell the account owner to fix it at the source site.
• LOGIN_PAGE_CHANGED: update the integration because the site changed its flow.

Wire returns explicit auth failure codes such as AUTH_EXPIRED, MFA_REQUIRED, and LOGIN_PAGE_CHANGED, which is the difference between a retryable session problem and a site integration problem.

What to build next

If you are adding authenticated scraping to an agent or workflow system, start by drawing the credential boundary. The worker should accept a sessionId, not a password. Store encrypted session state, refresh it when the site does, and return explicit auth errors when it stops working.

Then move the original credential into a vault and store only a reference to it. That one change removes the worst object from your application database.