Stop Comparing Proxy Vendors with One Success-Rate Number

Disclosure: I work with 98IP, a proxy infrastructure provider. This article is an engineering method for evaluating any vendor; it does not rank providers.

A dashboard says Vendor A succeeded on 92% of requests and Vendor B on 95%. The obvious conclusion is that B wins.

That conclusion may be wrong.

If the test sent only 200 requests, the observed rates are estimates, not permanent properties of either network. If A handled harder markets, if B received more cached pages, or if retries were counted as new observations, the comparison is weaker still.

This is the smallest evaluation framework I would accept before turning a residential-proxy test into a buying decision.

1. Define one trial before writing code

A trial should represent one business task, not one socket attempt. For example:

Fetch one authorized product page through one assigned route, validate the expected content, and finish within 15 seconds without an application retry.

Record the first attempt separately from later retries. Otherwise aggressive retry logic can make an unreliable route look healthy while quietly multiplying bandwidth and latency.

A useful row has at least:

{
  vendor: "candidate-a",
  market: "US",
  targetClass: "retail-product",
  sessionMode: "sticky-10m",
  startedAt: "2026-08-29T01:10:00Z",
  httpStatus: 200,
  contentValid: true,
  latencyMs: 1840,
  bytes: 91234,
  retryIndex: 0,
  failureLayer: null
}

Do not put raw proxy credentials, full query strings, cookies, or personal data in the result file.

2. Report uncertainty, not just the point estimate

For a binary outcome, a Wilson score interval is a practical way to show how uncertain the measured success rate is. This JavaScript implementation uses no dependencies:

function wilson(successes, trials, z = 1.96) {
  if (!Number.isInteger(successes) || !Number.isInteger(trials)) {
    throw new TypeError("successes and trials must be integers");
  }
  if (trials <= 0 || successes < 0 || successes > trials) {
    throw new RangeError("invalid binomial counts");
  }

  const p = successes / trials;
  const z2 = z * z;
  const denominator = 1 + z2 / trials;
  const center = (p + z2 / (2 * trials)) / denominator;
  const margin =
    (z * Math.sqrt((p * (1 - p) + z2 / (4 * trials)) / trials)) /
    denominator;

  return {
    observed: p,
    lower: center - margin,
    upper: center + margin
  };
}

console.log(wilson(184, 200));
// observed: 0.92, approximately 0.874 to 0.950 at 95% confidence

The lower bound is often more useful for procurement than the observed rate. If your acceptance floor is 90%, an observed 92% from 200 trials does not establish that requirement convincingly: the interval still includes values below 90%.

This is not a trick for declaring a winner. It is a guardrail against pretending a small test is precise.

3. Stratify before aggregating

One global rate can hide the failure that matters. Group results by dimensions that affect the real workload:

• target class;
• country or region;
• session mode;
• authentication path;
• protocol family;
• first attempt versus retry;
• time window.

Suppose a provider produces 97% success in North America and 71% in the one European market that generates most of your revenue. An attractive global average is not an operational pass.

Create the test matrix first, then randomize the provider order within each cell. That reduces the chance that one candidate receives the easy hour while another receives a target outage.

4. Compare latency as a distribution

Mean latency alone is easy to misread. Keep at least p50, p95, and p99 for successful first attempts, plus time-to-terminal-failure for failed trials.

Also separate:

• connection and TLS time;
• time to first byte;
• full transfer time;
• application validation time.

This tells you whether a slow result comes from the route, the destination, or your own parser.

5. Normalize cost by valid work

Price per GB is not a complete comparison. Use cost per valid success:

effective cost = total pilot cost / valid first-attempt successes

Include billable retries, duplicate payloads, failed transfers, and operational overhead. A cheaper rate card can become the expensive option when the workflow must repeat many requests.

6. Use a written decision gate

Before the first request, write the rule that produces pass, fail, or inconclusive. For example:

• every critical market has enough independent trials;
• the 95% lower confidence bound exceeds the market's minimum success threshold;
• p95 latency stays inside the service budget;
• effective cost remains below the ceiling;
• no critical compliance or credential-handling defect is open.

If the interval is too wide, the honest result is inconclusive. Collect more representative observations instead of changing the threshold after seeing the result.

A compact review function

function reviewCell({ successes, trials, minLowerBound, p95Ms, maxP95Ms }) {
  const interval = wilson(successes, trials);
  const enoughReliability = interval.lower >= minLowerBound;
  const fastEnough = p95Ms <= maxP95Ms;

  return {
    trials,
    observedPercent: +(interval.observed * 100).toFixed(2),
    lower95Percent: +(interval.lower * 100).toFixed(2),
    upper95Percent: +(interval.upper * 100).toFixed(2),
    p95Ms,
    decision: enoughReliability && fastEnough ? "pass" : "fail-or-extend"
  };
}

In production, validate the implementation against a trusted statistics package and keep the raw, sanitized trial rows so another reviewer can reproduce the result.

Final checklist

• One business task equals one first-attempt trial.
• Success includes content validation, not only HTTP status.
• Vendors receive the same randomized workload.
• Results are reported per critical segment.
• Confidence intervals accompany proportions.
• Latency is a distribution, not one average.
• Cost is normalized by valid output.
• The decision rule is frozen before testing.
• Credentials and personal data never enter the report.
• Testing is limited to authorized destinations and lawful purposes.

The full procurement worksheet and worked examples are available from 98IP.