Session Reuse and Browser Mode Decisions for Web Data Pipelines

A scraper that works on public HTML often fails the first time you point it at real customer data. Instead of the dashboard content, you index a login page. Instead of a product catalog, you get region-specific defaults. Instead of JSON, your parser throws because the page is just a JavaScript shell.

These are not parsing problems first. They are state and execution problems.

Know what kind of page you are fetching

Before reaching for a browser, classify the failure.

Common symptoms:

Symptom	What it usually means
401 Unauthorized	Missing or expired credentials
HTML contains Sign in or Log in	You fetched the login state, not the authenticated state
403 Forbidden	Blocked client, wrong region, or missing auth
Empty <div id="root"></div> page	Content renders client-side after hydration
Different price than expected	Geo-routing or account tier changed the response
Unexpected token < in JSON at position 0	Code expected JSON but received HTML, often an error page

Do this classification before adding retries. Retrying a login page gives you the same login page.

Reuse sessions only inside clear boundaries

Authenticated scraping needs session state: cookies, local storage, sometimes an assigned exit region. Re-authenticating for every fetch wastes time and can trigger security systems. Reusing sessions too broadly creates a worse problem: data leaks across users or tenants.

A reasonable policy is:

• Same domain, same user, same auth context: reuse the session.
• Same domain, different role or account: create a new session.
• Different customer or tenant: always create a new session.
• Logged-in and logged-out views: treat them as separate sessions.
• Expired or revoked credentials: delete the session and require re-authentication.

You can encode that policy directly instead of leaving it to caller discipline:

type SessionKeyInput = {
  tenantId: string;
  userId: string;
  domain: string;
  authContext: 'anonymous' | 'customer' | 'admin';
  country?: string;
};

function sessionKey(input: SessionKeyInput) {
  return [
    input.tenantId,
    input.userId,
    input.domain,
    input.authContext,
    input.country ?? 'default'
  ].join(':');
}

function shouldReuseSession(a: SessionKeyInput, b: SessionKeyInput) {
  return sessionKey(a) === sessionKey(b);
}

The key point is tenant isolation. If two customers use the same upstream SaaS app, they still need separate sessions. Shared cookies are a security bug, not an optimization.

Set an expiration policy too. Ninety days may be fine for some internal tools. Banking or healthcare workflows may need much shorter retention. Whatever you choose, make deletion explicit and irreversible from the application point of view.

Pick standard HTTP before browser automation

Browser mode costs more because it does more. It starts a browser context, runs JavaScript, waits for the page to settle, and sometimes interacts with the page. That is useful when the content only exists after client-side execution. It is wasteful when the server already returns usable HTML.

A good default decision tree looks like this:

type FetchMode = 'http' | 'browser';

type PageSignals = {
  status: number;
  html: string;
  needsScroll?: boolean;
  hasCaptcha?: boolean;
};

function chooseFetchMode(signals: PageSignals): FetchMode {
  if (signals.status === 403) return 'browser';
  if (signals.hasCaptcha) return 'browser';
  if (signals.needsScroll) return 'browser';

  const looksLikeJsShell =
    signals.html.includes('<div id="root"></div>') ||
    signals.html.includes('<div id="app"></div>');

  if (looksLikeJsShell && signals.html.length < 5000) {
    return 'browser';
  }

  return 'http';
}

This is intentionally conservative. A React app can still server-render useful HTML, and a short HTML document is not always a failure. Log the decision and sample responses so you can tune the heuristics per domain.

If your extraction provider can authenticate and extract at the network layer, Wire belongs on the non-browser side of this decision: use browser automation only when page behavior requires a real browser.

Keep geo-routing part of the session identity

Region-specific content creates subtle bugs. The request succeeds, the parser works, and the model still answers incorrectly because it saw the US catalog instead of the India catalog.

Treat country or region as part of the request and session identity:

{
  "url": "https://example.com/pricing",
  "session_id": "tenant_42:user_7:example.com:customer:in",
  "country": "in"
}

Do not reuse a country=us authenticated session for a country=in fetch unless you have verified the upstream application ignores region. Many sites bind catalog, currency, legal text, and inventory to both account and IP region.

Cache by everything that changes the response

Caching only by URL is usually wrong for authenticated or regional pages.

Use a cache key that includes the meaningful request dimensions:

import crypto from 'node:crypto';

function cacheKey(input: {
  url: string;
  tenantId: string;
  authContext: string;
  country?: string;
  headers?: Record<string, string>;
}) {
  const material = JSON.stringify({
    url: input.url,
    tenantId: input.tenantId,
    authContext: input.authContext,
    country: input.country ?? null,
    acceptLanguage: input.headers?.['accept-language'] ?? null
  });

  return crypto.createHash('sha256').update(material).digest('hex');
}

For public pages, a URL-only cache may be enough. For logged-in dashboards, pricing pages, or catalogs, include tenant, auth context, country, and any header that changes the response.

Monitor the failures that indicate wrong mode

Track more than success rate. A request that returns 200 OK with a login page is still a failed extraction.

Useful production checks include:

• Percentage of fetched pages that contain login markers.
• Percentage of pages with very low text length.
• Browser fallback rate per domain.
• Session expiration rate.
• Region mismatch reports from downstream validation.

If browser fallback climbs from 5% to 60% for a domain, something changed. Maybe the site moved rendering client-side. Maybe it started blocking your HTTP client. Either way, you want to notice before stale or empty documents enter the index.

The practical next step is to add response classification before parsing: detect login pages, JS shells, region mismatches, and empty content, then choose session reuse or browser execution based on those signals.

The full breakdown is here if you want the complete picture.