Find the Safe Proxy Concurrency Ceiling for Browser Automation

#webdev #automation #testing #performance

Browser concurrency and proxy capacity are different limits. A platform may launch 200 browser sessions while the proxy gateway, regional inventory, destination policy, or result pipeline saturates much earlier.

This test plan finds the highest useful throughput without hiding failures behind retries.

Define the result model

type StageResult = {
  concurrency: number;
  launched: number;
  firstAttemptPassed: number;
  retryRecovered: number;
  validated: number;
  p50Ms: number;
  p95Ms: number;
  proxyAuthFailures: number;
  rateLimited: number;
  bytes: number;
  cost: number;
};

function usefulPerMinute(r: StageResult, elapsedMs: number) {
  return r.validated / (elapsedMs / 60_000);
}

Do not merge firstAttemptPassed and retryRecovered. They answer different questions.

Run stepped load

Use stages such as 10, 25, 50, 100, 150, and 200. Change only concurrency while keeping the region, page cohort, proxy type, browser version, and validation rules fixed.

For every stage measure:

launch and proxy-authentication success;
connection, TLS, DOM-ready, and workflow time;
p50 and p95 completion time;
403, 407, 429, and 5xx outcomes by cause;
unexpected location or sticky-session changes;
queue wait and cleanup time;
bytes and cost per validated result;
CPU, memory, file descriptors, and database saturation.

Stop increasing load when throughput flattens, p95 breaches the objective, or error rates rise across two consecutive stages.

Put backpressure before retry

import pLimit from 'p-limit';

const newWork = pLimit(80);
const retryWork = pLimit(8);

async function runWithBudget<T>(
  first: () => Promise<T>,
  retry: () => Promise<T>,
) {
  try {
    return await newWork(first);
  } catch (error) {
    if (!isIdempotent(error)) throw error;
    await jitter(400, 1_200);
    return retryWork(retry);
  }
}

The exact numbers should come from the benchmark. The principle is stable: retry capacity must be materially smaller than first-attempt capacity.

Separate browser workers from proxy sessions

Use one proxy session per context when journeys require strong isolation.
Keep one sticky session for an authorized multi-step workflow that requires location continuity.
Rotate per independent request only when the request is truly stateless.

Never share sticky sessions across unrelated users or jobs. Keep credentials out of logs, traces, screenshots, and example code.

Choose a production ceiling

Start below the observed saturation point—often around 70–80% of the lowest verified threshold—and reduce automatically when latency, error, or spend budgets are exceeded.

The deployment record should include region-specific concurrency, proxy-session rules, queue limits, retry budget, p95 target, first-attempt success target, cost ceiling, and stop conditions.

Only test systems and data you are authorized to access. Respect destination terms, rate limits, privacy obligations, and regional requirements. Do not use proxy rotation or concurrency to bypass access controls.

Disclosure: I work with 98IP. More practical proxy-testing guidance: https://en.98ip.com/?k=dev

DEV Community