DEV Community: 98IP Proxy

Find Proxy Throughput Before Retries Find You

98IP Proxy — Thu, 03 Sep 2026 21:15:27 +0000

Find Proxy Throughput Before Retries Find You

Doubling workers does not guarantee twice as many useful pages. A proxy-backed pipeline can saturate its own connection pool, gateway, browser fleet, authorized destination allowance, parser, or downstream queue. Retries then create more work while the useful output curve goes flat.

I work with 98IP. This is the compact ramp model I would use before buying more capacity—including from us.

Define the output before the load

type StageResult = {
  concurrency: number;
  originalJobs: number;
  totalAttempts: number;
  validResults: number;
  elapsedMs: number;
  queueP95Ms: number;
  latencyP95Ms: number;
  rateLimited: number;
  proxyAndComputeCost: number;
};

function score(s: StageResult) {
  return {
    usefulPerMinute: s.validResults / (s.elapsedMs / 60_000),
    retryAmplification: s.totalAttempts / s.originalJobs,
    costPerValid: s.validResults
      ? s.proxyAndComputeCost / s.validResults
      : Infinity,
  };
}

Raw requests per second is a transport metric. usefulPerMinute is closer to a business capacity metric because it requires validated output.

Ramp one variable set at a time

Hold the client build, proxy product, region, protocol, address family, session policy, target set, timeouts, retry rules, and parser contract constant.

Then run safe stages such as 1, 2, 4, 8, 12, and 16 workers—only inside the limits of systems you own or are authorized to test. Warm each stage, observe long enough to include normal latency, and repeat before promotion.

Stop when any gate fails. Do not jump straight to the contractual maximum.

Keep first attempts visible

Measure content-valid first attempts before retries. A status 200 can still be a login shell, empty table, regional variant, stale page, or block response.

Classify connection, proxy authentication, DNS, TLS, header timeout, 429/503, content contract, region/session, parser, and storage failures separately.

If the service sends Retry-After, respect it as a minimum delay for that scope. Do not rotate identities to defeat a rate limit. Eligible transient retries need exponential backoff with jitter, an attempt ceiling, and a batch-level budget.

Watch queue age

One slow target can occupy a shared worker pool while dashboards report high utilization. Track queue depth and p95 age by target class. Add per-target concurrency caps or isolated queues so one domain cannot starve unrelated authorized work.

Select the point before the knee

Promote a stage only while:

useful throughput rises materially;
first-attempt validity remains inside the gate;
p95 latency and queue age stay bounded;
retry amplification stays near baseline;
429/503 does not trend upward;
region and session accuracy stay valid;
cost per valid output stays flat or improves.

The next stage after these relationships break is the knee. Set production below it and retain headroom.

Normalize providers with one question: What will this exact authorized workload cost per validated result inside its freshness window? Per-GB, per-request, and per-thread prices are otherwise difficult to compare.

Use proxies only for systems and data you are authorized to access. Respect contracts, access policies, rate limits, privacy obligations, and applicable law. Concurrency and rotation must never be used to evade controls or misrepresent identity.

Disclosure: I work with 98IP. More proxy engineering guidance: https://en.98ip.com/?k=dev

Your Monthly Browser-Proxy Test Window Just Became Too Slow

98IP Proxy — Thu, 03 Sep 2026 19:15:18 +0000

Your Monthly Browser-Proxy Test Window Just Became Too Slow

Chrome 153 is scheduled to begin Chrome's two-week stable milestone cadence on September 8. If a browser automation team still validates the browser-plus-proxy stack once a month, it can spend much of every cycle certifying yesterday's build.

I work with 98IP. The practical response is not “test everything twice as often.” It is to define a small support window and make every result reproducible.

Model the intersection, not the components

A direct browser check does not cover an authenticated proxy path. At minimum, the release gate should identify:

type ProxyBrowserCase = {
  browserBuild: string;
  driverBuild: string;
  osImage: string;
  gatewayProduct: "rotating-residential" | "sticky-residential" | "isp";
  protocol: "http-connect" | "socks5";
  requestedRegion: string;
  addressFamily: "ipv4" | "ipv6";
  sessionMode: "rotating" | "sticky";
  targetContract: string;
};

Avoid browserBuild: "latest". A failed run with an unknown binary cannot be compared or rolled back cleanly.

Keep the support window small

For each production host family, test only:

current production stable and its matching driver;
next beta and its matching driver;
previous stable while it remains an approved rollback;
proxy protocols and address families actually used by production;
one controlled target plus one representative authorized workload.

This is a rolling window, not a historical museum.

Make “valid result” stricter than status 200

type ProbeResult = {
  caseId: string;
  startedAt: string;
  tunnelOk: boolean;
  tlsOk: boolean;
  contentContractOk: boolean;
  regionOk: boolean;
  stickySequenceOk: boolean | null;
  firstAttempt: boolean;
  latencyMs: number;
  bytes: number;
  errorClass: string | null;
};

function accepted(r: ProbeResult) {
  return r.tunnelOk &&
    r.tlsOk &&
    r.contentContractOk &&
    r.regionOk &&
    r.firstAttempt &&
    r.latencyMs < 5000;
}

A consent page, login shell, regional redirect, or block page can all be HTTP-successful. Assert the title, schema, locale, region marker, or navigation state the job actually requires.

Promote in stages

Run beta canaries against systems you own or are allowed to test. Then promote a new stable build through 5%, 25%, 50%, and 100% of authorized traffic. Keep the previous image available until the observation window closes.

Use failure groups that map to action:

launch or driver attachment;
proxy authentication or CONNECT;
DNS and address-family selection;
TLS or certificate handling;
rendering or navigation;
content contract;
geography or session continuity;
capacity, latency, or retry amplification.

When a regression appears, compare the previous browser, direct path, known-good gateway, rotating versus sticky mode, and IPv4 versus IPv6—one variable at a time.

Do not let retries hide a bad release

Report first-attempt useful-result rate separately from retry-recovered success. Otherwise an incompatible build can look healthy while consuming more proxy bandwidth, destination capacity, and time.

The release gate should keep the same thresholds even when releases arrive faster. Shorter cadence calls for smaller automated tests, not weaker acceptance criteria.

Use browser automation and proxies only on systems and data you are authorized to access. Respect destination terms, rate limits, privacy duties, provider restrictions, and applicable law.

Disclosure: I work with 98IP. More proxy testing guidance: https://en.98ip.com/?k=dev

ASN Data Cannot Prove a Residential Proxy Is Residential

98IP Proxy — Thu, 03 Sep 2026 17:22:39 +0000

ASN Data Cannot Prove a Residential Proxy Is Residential

A proxy exits through an ASN with a familiar broadband-carrier name. Is that proof that the route is residential?

No. It is useful evidence about routing context, but it does not prove physical access type, device location, participant consent, exclusivity, reputation, or how a destination will classify the address.

I work with 98IP. This is the audit model I would want a buyer to use before approving any residential proxy product—including ours.

Keep five evidence layers separate

Represent every sampled exit as a record with distinct fields:

type ExitAudit = {
  sampleId: string;
  observedAt: string;
  requested: {
    product: "rotating-residential" | "static-isp" | "mobile" | "datacenter";
    country: string;
    addressFamily: "ipv4" | "ipv6";
    sessionMode: "rotating" | "sticky";
  };
  routing: {
    asn: number | null;
    prefix: string | null;
    rdapOrganization: string | null;
  };
  classifications: Array<{
    source: string;
    value: "residential" | "isp" | "mobile" | "datacenter" | "unknown";
    checkedAt: string;
  }>;
  provenance: {
    productDocumentVersion: string | null;
    consentMechanismReviewed: boolean;
    revocationProcessReviewed: boolean;
    upstreamControlsReviewed: boolean;
  };
  outcome: {
    tunnelOk: boolean;
    contentValid: boolean;
    regionValid: boolean;
    firstAttempt: boolean;
    latencyMs: number;
  };
};

The structure prevents a common analytical shortcut: allowing a broadband-sounding ASN to overwrite missing provenance evidence.

Score disagreement, not “truth”

Network-type databases can disagree. Do not average their labels into a fake consensus. Preserve every observation and compute a disagreement rate:

function classificationSummary(row: ExitAudit) {
  const labels = row.classifications.map(x => x.value);
  const expected = new Set(["residential", "isp"]);
  const expectedVotes = labels.filter(x => expected.has(x)).length;
  const known = labels.filter(x => x !== "unknown").length;

  return {
    sources: labels.length,
    known,
    expectedRate: known ? expectedVotes / known : null,
    unanimous: new Set(labels.filter(x => x !== "unknown")).size <= 1,
  };
}

This still does not certify provenance. It only describes how classification sources treated the sample at specific times.

Add concentration to the audit

Thirty sampled exits may look diverse while 80% of the traffic comes from one ASN. Measure shares by region and session mode:

function largestShare(values: Array<string | number | null>) {
  const usable = values.filter((v): v is string | number => v !== null);
  const counts = new Map<string | number, number>();
  for (const value of usable) counts.set(value, (counts.get(value) ?? 0) + 1);
  return usable.length
    ? Math.max(...counts.values()) / usable.length
    : null;
}

Run this for ASNs and for a consistently defined prefix bucket. A large advertised pool does not guarantee that your workload sees diverse networks in the countries you buy.

Provenance needs documents and controls

Ask how devices or networks enter the exact product, what participants see before opting in, how they revoke participation, how fast removal occurs, whether upstream suppliers are involved, and how the same requirements are audited downstream.

Also review abuse handling, destination restrictions, metadata retention, security ownership, and the date and scope of any independent assurance.

The result is not a binary “ethical” badge. It is a product-specific evidence package with an owner and an expiry date.

Test the workload separately

Only use targets you own or are authorized to test. For every sample, validate the content contract—not only HTTP status. Record the requested and observed region, first-attempt result, latency, session continuity, and cost per valid result.

Keep retry recovery separate. Otherwise a weak route can look successful after consuming extra bandwidth and target capacity.

Use four decisions

Pass: documentation and sampled behavior meet the predefined gates.
Conditional pass: only named products, regions, or traffic levels are approved.
Fail: repeated contradictory clusters or missing required controls remain unresolved.
Inconclusive: the sample or documents cannot support a decision.

Inconclusive should trigger better evidence, not automatic approval.

Use proxies only for authorized systems and data. Respect provider contracts, destination rules, rate limits, privacy requirements, and applicable law. Network-type analysis must not be used to evade controls or misrepresent identity.

Disclosure: I work with 98IP. More proxy testing guidance: https://en.98ip.com/?k=dev

Estimate Proxy Bandwidth From Valid Outcomes, Not Page Size

98IP Proxy — Wed, 02 Sep 2026 21:12:59 +0000

When a proxy plan is priced by gigabytes, it is tempting to multiply “average page size” by request count. That estimate usually misses redirects, failed bodies, retries, uploads, headers, browser assets, cache behavior, and the gap between an HTTP 200 and a result your application can actually use.

Here is a small capacity model that is much harder to fool.

Start with the denominator that matters

Define a valid outcome first: the required fields are present, the market and locale are correct, freshness passes, and the response is not a block page or consent screen.

Then calculate each workload cell separately:

monthly_bytes = valid_outcomes
              × attempts_per_valid_outcome
              × billable_bytes_per_attempt

capacity_bytes = monthly_bytes
               × (1 + growth_rate)
               × (1 + safety_headroom)

Measure attempts_per_valid_outcome during a pilot:

attempts_per_valid_outcome = all_billable_attempts / valid_outcomes

That ratio reveals retry amplification without pretending every failed request is independently recoverable.

Keep three byte counters

Use matching request IDs and time windows to compare:

Application payload bytes — what the app consumed after decoding.
Client transfer bytes — what curl, the HTTP library, or the browser reports.
Provider-billed bytes — what the proxy dashboard counts.

They are not interchangeable. The application may omit headers and failed attempts. Automatic decompression can make the in-memory response larger than the transferred body. A provider may count uploads, redirects, error bodies, or both directions of a tunnel.

The contract's billing definition should drive the forecast; the other counters help explain mismatches.

Never mix browser and API averages

An API request might transfer a small JSON response. A browser visit can load HTML, scripts, styles, fonts, images, video, analytics, API calls, service-worker updates, and speculative resources.

Measure cold-cache and warm-cache browser sessions separately. Isolated profiles and rotating workers may not keep the cache savings you see on a developer laptop.

Block only resources proven unnecessary for the authorized output. Breaking consent, security, or application behavior to save bandwidth produces an invalid benchmark.

Use a range, not one number

For each target × region × client × proxy product × session mode cell, create low, expected, and high assumptions for:

attempts per valid outcome;
billable bytes per attempt;
monthly valid outcomes.

Example:

Scenario	Outcomes	Attempts/outcome	MB/attempt	Base GB
Low	200,000	1.05	0.20	42
Expected	200,000	1.18	0.24	56.64
High	200,000	1.45	0.32	92.8

With 15% growth and 20% safety headroom, the expected capacity is about 78.16 GB.

Those are example numbers, not defaults. A browser workflow can be orders of magnitude larger.

Calibrate before committing

Run a small authorized pilot:

representative targets for every critical cell;
concurrency one at first;
retries disabled for the baseline;
planned bounded retries for the second run;
cold and warm cache separated;
application, client, and provider counters reconciled;
403 and 429 treated as stop signals, not rotation triggers.

Extend the pilot when a critical cell still has an unstable range. A global average can hide the one small market that drives the purchase.

Finally, review overage pricing, rollover, expiration, minimum commitment, and whether the vendor calls a billion bytes a GB or uses GiB.

I work with 98IP, a proxy service. I am sharing this because forecasting from successful outcomes makes plan comparisons more honest and reduces retry waste. Product information: https://en.98ip.com/?k=dev

Use proxies only for authorized systems and data. Follow destination terms, robots guidance, rate limits, privacy obligations, and data-minimization requirements. Do not budget traffic for repeatedly evading denials.

curl 8.22 Fixed an HTTP/3 Proxy Crash. Test the Boundary, Not Just the Happy Path

98IP Proxy — Wed, 02 Sep 2026 19:14:50 +0000

curl 8.22.0 shipped on September 2 with two proxy-relevant fixes: an HTTP/3 proxy path no longer dereferences a null pointer when a non-status response header arrives before :status, and CONNECT trailer handling was corrected.

If you operate proxy-backed jobs, the useful lesson is not “turn on more retries.” It is that your test suite needs to distinguish a client parser failure from a gateway failure, exit failure, and destination response.

How a client bug becomes fake proxy churn

Imagine a worker crashes while parsing an unusual HTTP/3 proxy response. The scheduler sees an incomplete job, selects a different exit, and retries. The same client hits the same condition and crashes again.

Your dashboard now shows several failed exits, but the exit was never the independent variable. Rotation multiplied the symptom.

CONNECT framing errors can produce a quieter version of the same mistake: a tunnel is labelled unavailable even though authentication and routing completed. That can degrade a healthy gateway cohort and increase cost per successful request.

A compact paired test

Run the old and new curl builds against the same authorized fixture. Hold the gateway, credentials, protocol backend, target, concurrency, timeouts, and rotation policy constant.

Include six cases:

Normal HTTP/1.1 CONNECT.
Your supported HTTP/2 proxy path.
HTTP/3 proxy handling in a controlled fixture.
A controlled CONNECT response with trailers.
Invalid proxy credentials as a negative control.
An unreachable gateway to prove fail-closed behavior.

For each run, capture the curl version and linked backend, negotiated protocol, sanitized gateway cohort, curl error code, process exit or crash signal, tunnel outcome, total latency, retry count, and application-level validity.

Do not store proxy credentials, cookies, authorization headers, or unnecessary payloads in the artifact.

Classify at the first failure boundary

Process crash: quarantine the client build or test condition. Do not rotate exits.
Protocol parser error: preserve a sanitized diagnostic and reproduce against a controlled server.
407 or explicit proxy denial: inspect authentication and gateway policy.
Tunnel timeout: inspect gateway reachability, TLS to the proxy, and timeout phase.
403/429 from the destination: respect the response; do not treat rotation as a fix.
200 with invalid content: transport succeeded. Validate locale, redirect, cache, and schema separately.

This one change in taxonomy prevents a client problem from corrupting pool-health data.

Canary metrics that matter

Start with one worker or no more than one percent of eligible traffic. Compare old and new builds by protocol and gateway cohort:

crashes per 10,000 attempts;
successful tunnels per attempt;
retries per valid result;
unique exits consumed per valid result;
p50/p95 connection time;
categorized curl errors;
application-valid responses per attempt.

Roll back if crash rate is nonzero, direct fallback appears, authentication errors rise unexpectedly, or cost per valid result worsens materially. Adding retries is not a rollback plan.

One important limit

A curl fix does not prove your proxy provider supports HTTP/3. Support still depends on how curl was built, the HTTP/3 backend, the proxy service, and the network path. Keep malformed-response tests inside infrastructure you control or are authorized to test.

I work with 98IP, a proxy service. I am sharing this because client-side failures are routinely mislabelled as “bad proxies,” which makes both reliability and cost analysis worse. For product details: https://en.98ip.com/?k=dev

Use proxies only where authorized. Follow destination terms, rate limits, robots guidance, privacy rules, and data-minimization requirements. Do not use retries or rotation to evade access controls.

Official basis: curl project, “Changes in 8.22.0,” September 2, 2026.
``

Stop Comparing Proxy Plans by Price per GB

98IP Proxy — Wed, 02 Sep 2026 17:14:15 +0000

The least expensive proxy plan on a pricing page can be the most expensive route in production.

Price per GB and price per IP describe what you buy. They do not describe what the workload actually delivers. Retries consume traffic, slow responses occupy workers, block pages pass superficial status checks, and invalid records create repair work later. A better procurement metric is cost per validated successful request.

This is the scorecard I would use for an authorized crawler, monitoring job, ad-verification system, or market-research workflow.

Start with a business-level success definition

Do not count every HTTP 200 as success. Define the result that downstream users can actually consume.

A request might count only when all of these are true:

the proxy connection and TLS negotiation complete;
the response status and content type are expected;
the body is not a challenge or error template;
required fields are present and fresh;
the requested country or session behavior is correct;
latency remains inside the job deadline;
the access is authorized and consistent with the destination's rules.

The definition should be machine-checkable. If an analyst has to inspect every response manually, the test is measuring operator patience as much as proxy quality.

Calculate the full numerator

Use this basic equation:

cost_per_validated_success = total_workload_cost / validated_successes

The numerator should include more than the proxy invoice:

proxy bandwidth, IP, port, or subscription charges;
bandwidth spent on failed attempts and permitted retries;
browser or worker compute time;
parsing, validation, storage, and observability costs;
engineering time used to investigate failures;
manual repair required before data becomes usable.

Suppose Plan A costs $100 and produces 9,000 validated results with $25 of compute and operations. Its unit cost is about $0.0139. Plan B costs only $75 but produces 5,000 valid results and creates $40 of additional work. Its unit cost is $0.023. The cheaper invoice produced a result that was roughly 65% more expensive.

Instrument the request funnel

A single success percentage hides where money disappears. Record at least five stages:

attempted
  -> transport response received
  -> acceptable HTTP/content response
  -> validation passed
  -> usable downstream outcome

The gaps have different meanings.

Attempted to transport response: inspect DNS, proxy negotiation, TLS, timeouts, and gateway capacity.
Transport to acceptable response: inspect status classes, content types, and destination backpressure.
Acceptable response to validation: detect challenge templates, missing fields, stale pages, and locale errors.
Validation to usable outcome: inspect parsers, schemas, deduplication, and downstream delivery.

This separation prevents a parser regression from being blamed on the proxy and prevents a weak route from being hidden behind aggressive retries.

Keep the comparison controlled

Run competing plans against the same representative workload. Keep these variables fixed:

target set and request method;
country and city requirements;
concurrency and time-of-day window;
timeout and retry budget;
static or rotating session policy;
browser or client version;
validation rules.

Randomize request assignment so one provider does not receive only easy targets. Report first-attempt success separately from final success after retries. Otherwise a plan can appear reliable while consuming two or three times the traffic.

Segment before choosing a winner

Global averages are dangerous. Calculate unit cost by region, destination, response size, session mode, and concurrency band.

A pool might be excellent for small North American API responses and inefficient for large APAC pages. Another might have a slightly higher blended price but far fewer retries in the one market that matters to the business. Routing workloads by measured strengths is often better than declaring one universal winner.

Add stop conditions to the pilot

Write procurement gates before the test begins. For example:

validated success must remain above the required threshold;
p95 latency must fit the job deadline;
retries per usable result must stay below a fixed budget;
location mismatches must remain below an acceptable rate;
unit cost must remain below the business ceiling;
no test may bypass access controls or continue after explicit backpressure.

Predefined gates make it harder to rationalize a poor result because a plan looked attractive on the pricing page.

A compact scorecard

For every candidate, capture:

attempted requests;
first-attempt validated successes;
final validated successes;
bytes transferred;
retry count by reason;
median and p95 latency;
route or location mismatches;
challenge and invalid-content rate;
compute and operator time;
total cost and cost per validated success.

Re-run the same scorecard periodically. Proxy performance, destination behavior, traffic mix, and your own client code all change.

The goal is not the cheapest connection. It is the lowest sustainable cost for a result the business can trust.

Disclosure: I work with 98IP. This article reflects an operator perspective and does not claim that a proxy permits access to systems or data you are not authorized to use. Respect destination policies, rate limits, privacy obligations, and applicable law. More English proxy engineering resources are available at https://en.98ip.com/?k=dev.

Rotate Proxy Credentials Without Turning Rotation Into an Outage

98IP Proxy — Wed, 02 Sep 2026 13:11:41 +0000

Proxy credentials are often treated like a configuration value: replace the old secret, restart the workers, and move on.

That is exactly how a routine security task becomes a production incident.

A crawler, monitoring agent, or data-collection service rarely has one credential consumer. The secret may exist in a scheduler, container platform, local development profile, serverless job, browser worker, health check, and emergency runbook. Those consumers do not all reload at the same time. A safe rotation therefore needs overlap, observability, and a rollback boundary.

This guide describes a provider-neutral rollout for systems you own or are authorized to operate.

First, separate the credential from proxy policy

Treat these as different configuration objects:

gateway hostname and port;
authentication username or account identifier;
secret or token;
target country, region, or session parameters;
retry and timeout policy.

If all five values live in one opaque URL, a password change can accidentally change routing behavior too. Parse the connection configuration once, keep the secret in a secret manager, and inject only the current credential at runtime.

Never print the full proxy URL. A masked log such as http://user:***@gateway:port is still useful for identifying the gateway without turning logs into a credential store.

Use an overlap window

The safest model is two valid credentials for a short, controlled period:

Create the new credential without revoking the old one.
Deploy the new secret to a small canary group.
Verify authentication, tunnel creation, TLS completion, DNS behavior, target-region accuracy, and session stability.
Expand the rollout in stages.
Confirm that no active workload is using the old credential.
Revoke the old credential.

If your provider cannot support overlapping credentials, schedule a maintenance window or use a blue/green account pattern. Do not assume that a synchronized restart will be truly simultaneous across queues, autoscalers, and long-running jobs.

Make the client reloadable

A long-running worker should be able to obtain the current secret without a full application redeploy. The exact mechanism depends on your platform, but the design should have three properties:

the secret is read from a managed source;
workers can refresh it on a bounded interval or configuration event;
in-flight requests keep their original credential while new requests use the new one.

Avoid retrying an authentication failure forever. A 407 Proxy Authentication Required after rotation is usually a configuration signal, not evidence that you need another exit IP.

Here is a language-neutral decision model:

if response == 407:
    refresh_secret_once()
    retry_once_with_same_route()
    if response == 407:
        open_rotation_circuit()
        alert("credential rejected")

The same route matters. Rotating the exit at the same time changes two variables and makes diagnosis harder.

Canary on more than status code

A successful TCP connection is not enough. For each canary, record:

secret version identifier, never the secret value;
worker release and configuration revision;
gateway and requested region;
proxy authentication result;
CONNECT or SOCKS handshake result;
DNS mode and negotiated IP family;
TLS result;
exit geography and ASN, when that is part of the contract;
request latency and application result.

Compare the new credential with a known-good control under the same workload. A sudden increase in 407 responses points toward distribution or activation problems. Normal authentication with changed geography points toward a routing configuration change. Normal routing with application rejection points farther downstream.

Roll out in bounded stages

A practical sequence is 1%, 10%, 50%, and 100% of workers. Hold each stage long enough to observe at least one normal job cycle.

Define stop conditions before deployment. Examples:

more than 0.5% proxy-authentication failures;
any region mismatch for a regulated workflow;
p95 tunnel latency increasing by more than 20%;
workers continuing to request the retired secret version;
queue growth beyond the normal retry budget.

When a stop condition triggers, pause expansion. Do not compensate with unlimited retries: that can amplify load and hide the actual distribution failure.

Prove the old credential is unused

Revocation should be based on evidence, not elapsed time alone. Check:

secret-manager access logs by version;
active worker configuration versions;
scheduled and paused jobs;
disaster-recovery environments;
local or CI profiles that can reach production;
health checks and synthetic monitors.

Then run a negative test from an isolated, authorized environment: the old credential should fail, while the new credential should succeed. Do not place either secret in a ticket, chat transcript, screenshot, or HAR file.

Rotation checklist

[ ] Inventory every credential consumer.
[ ] Separate authentication from routing policy.
[ ] Create a new secret version and record its owner and expiry.
[ ] Confirm an overlap or maintenance strategy.
[ ] Canary with the same route and workload.
[ ] Monitor authentication, routing, latency, and application signals separately.
[ ] Roll out in bounded stages with written stop conditions.
[ ] Verify that the old version has no readers.
[ ] Revoke the old credential and run a negative test.
[ ] Remove temporary access and shorten retained diagnostic data.

The operational principle

Credential rotation is successful only when the new secret is accepted, the intended proxy route remains correct, every consumer has migrated, and the old secret is demonstrably unusable.

That makes rotation a small deployment—not a string replacement.

Disclosure: I work with 98IP. This article reflects an operator perspective and does not promise that credentials can override a destination's access policies. Use proxy infrastructure only for systems and data you are authorized to access. More English proxy engineering resources are available at https://en.98ip.com/?k=dev.

A Happy Eyeballs Test Can Measure the Wrong Half of Your Proxy Route

98IP Proxy — Tue, 01 Sep 2026 22:15:07 +0000

Disclosure: I work with 98IP. This is vendor-neutral guidance for authorized connectivity testing.

Happy Eyeballs reduces the delay users feel when either IPv6 or IPv4 is slow. But once a proxy gateway is added, a single “IPv6 won” metric can describe the wrong connection.

There are at least two legs:

client --(IPv4 or IPv6)--> proxy gateway --(IPv4 or IPv6)--> destination

The client usually races addresses for the gateway. The gateway may resolve and connect to the destination independently. An IPv6 gateway connection can therefore produce an IPv4 exit, and an IPv4 gateway can use an IPv6 target path.

The IETF HAPPY Working Group's July 2026 Happy Eyeballs v3 draft is still an active Internet-Draft, not a final standard. Its work on scheduling, DNS results and remembered outcomes is a useful reason to make proxy measurement explicit now.

Observe four layers

DNS

Record who resolves the gateway and destination, the A/AAAA answer sets, order, resolver region, cache state and time. SOCKS hostname mode can move target DNS to the gateway; a client-resolved SOCKS request does not.

Gateway connection

Record the first and second attempt start times, gateway family, handshake result, TLS, proxy authentication, winner and cancellation delay. This is the leg a local Happy Eyeballs implementation usually controls.

Tunnel

Record CONNECT/SOCKS status, remote-DNS mode, target family when observable, tunnel time and a structured category such as:

gateway_unreachable
proxy_auth_failed
target_connect_failed
tunnel_timeout

Application result

Record first byte, complete response time, useful-result status and bytes. Keep destination policy results such as 403 and 429 out of the gateway transport score unless evidence connects them to the gateway.

Build a bounded matrix

Use a gateway and destination you own or have explicit permission to test:

IPv6-only gateway, remote target resolution.
IPv4-only gateway, same target and request.
Dual-stack gateway with the production racing logic.
Dual-stack gateway with a client-resolved destination control.
Preferred-family delay or loss injected in controlled infrastructure.
Both families healthy, using cold and warm connection pools.

Keep credentials, region, target, payload, timeout and workload purpose constant. Change one variable per comparison.

The useful metrics are not just connection time:

winning gateway family;
observable target family;
p50/p95 first byte;
tunnel and complete-response success;
mid-session reset rate;
fallback rate and fallback success;
extra connection attempts per useful result;
traffic and cost per authorized useful result.

Failure injection must not duplicate the request

A racing algorithm should race connection setup, not execute a purchase, message, upload or other state-changing action twice. Use an idempotent authorized canary, cap attempts and verify losing sockets, timers, pages and sessions are cleaned up.

Do not convert a target 429 into faster IP rotation. Do not claim end-to-end IPv6 because the client reached an IPv6 gateway. And do not penalize a provider for a destination policy decision.

Procurement questions

Ask whether the provider offers dual-stack, IPv4-only and IPv6-only gateway controls; where SOCKS hostname DNS runs; whether target family appears in sanitized logs; whether authentication, allowlists, concurrency and pricing differ by family; and whether direct fallback is impossible after gateway failure.

A public exit-IP check is useful, but it does not explain both routing decisions.

For more practical proxy and Internet-operations guidance, visit 98IP.

Operate only on authorized gateways, destinations, accounts and data. Respect website terms, robots directives, privacy obligations and rate limits. Address-family racing is a reliability technique, not a way to bypass a publisher's decision.
``

Stop Rotating Proxies When a Crawl Fails a Content-Use Check

98IP Proxy — Tue, 01 Sep 2026 20:18:56 +0000

Disclosure: I work with 98IP. This is a vendor-neutral operational pattern for authorized data collection.

Cloudflare's Browser Rendering /crawl endpoint now respects the Content Signals use directive. The caller can declare contentUse as reference or full; when the target's published ceiling is stricter, the request is rejected.

The most important engineering consequence is not the new parameter. It is the need to stop treating every rejected crawl as a bad proxy.

Add a policy result to your taxonomy

A useful classifier should separate at least:

proxy_auth_failed
dns_failed
tunnel_failed
transport_timeout
rate_limited
bot_challenge
content_use_denied
application_not_authorized
render_failed
success

content_use_denied is deterministic policy feedback. Rotating the exit, changing the user agent or retrying at higher concurrency does not repair the mismatch.

Keep four layers in every observation

{
  "identity": {
    "crawler": "catalog-reference-bot",
    "version": "2026.09.02"
  },
  "permission": {
    "approved_purpose": "reference",
    "declared_content_use": "reference",
    "observed_policy_at": "2026-09-02T03:00:00Z"
  },
  "transport": {
    "route": "us-east-residential-a",
    "family": "ipv4",
    "session_mode": "sticky"
  },
  "outcome": {
    "category": "content_use_denied",
    "retry": false,
    "bytes": 0
  }
}

Do not put proxy passwords, cookies, tokens or signing keys in this record.

Identity tells you who called. Permission tells you what the caller was allowed to do. Transport tells you how the request traveled. Outcome tells you what happened. Combining them into proxy_failed=true destroys the evidence needed for a correct response.

Retry by cause, not by status code

A safe retry function can be boring:

function retryDecision(result) {
  if (result.category === 'content_use_denied') return 'stop';
  if (result.category === 'application_not_authorized') return 'stop';
  if (result.category === 'proxy_auth_failed') return 'inspect_config';
  if (result.category === 'rate_limited') return 'respect_retry_after';
  if (result.category === 'transport_timeout' && result.attempt < 2) {
    return 'bounded_retry';
  }
  return 'stop';
}

The exact logic depends on your workload, but policy denial should never enter a proxy-rotation loop.

Test with an authorized policy canary

Before production rollout:

Use a site you own or have explicit permission to test.
Publish controlled content-use variants.
Send bounded reference and full jobs.
Confirm compatible jobs proceed.
Confirm incompatible jobs produce content_use_denied.
Verify no proxy rotation occurs after that result.
Check that pages, sessions and timers are cleaned up.

This canary validates policy logic without generating unwanted traffic against third-party sites.

Keep provider scoring independent

Measure proxy quality only on permitted workloads and controlled targets:

connect and TLS success;
time to first byte;
p50 and p95 latency;
throughput;
unexpected rotation;
address-family behavior;
ASN and prefix concentration;
cost per useful authorized result.

A destination rejecting a declared purpose is not evidence that the exit is slow, unstable or incorrectly located.

Operational checklist

Every crawler has a named purpose and owner.
contentUse is set from approved workload metadata.
Policy evidence is timestamped.
Policy denials have a no-retry rule.
Provider metrics exclude deterministic policy denials.
Diagnostic records exclude credentials and personal data.
Changes are tested on an authorized canary.
Robots directives, terms, privacy and rate limits remain separate gates.

What not to do

Do not silently relabel a full workload as reference. Do not rotate residential exits to search for a policy inconsistency. Do not report policy denials as an IP-quality defect. And do not assume that one compatible machine-readable signal grants permission for every purpose.

For more vendor-neutral proxy operations guidance, visit 98IP.

Proxies are a transport tool for authorized work, not a method for overriding publisher decisions.

Validate Residential Proxy Location as a Dataset, Not a Screenshot

98IP Proxy — Tue, 01 Sep 2026 18:12:31 +0000

Disclosure: I work with 98IP. This post describes a vendor-neutral test method and does not claim that one lookup service is authoritative.

A proxy gateway accepts country=US&city=New_York. Your first request returns an IP that one lookup service calls New York. Is the location requirement validated?

No. You validated one exit against one dataset at one time.

A production-quality acceptance test needs a row per exit observation and enough dimensions to expose pool-level inconsistencies.

Use an append-only observation schema

Start with a record like this:

{
  "run_id": "geo-us-nyc-001",
  "observed_at_utc": "2026-09-01T00:00:00Z",
  "requested": {
    "country": "US",
    "region": "NY",
    "city": "New York",
    "gateway": "us-east",
    "protocol": "http-connect",
    "address_family": "ipv4",
    "session_mode": "rotating"
  },
  "network": {
    "exit_ip_hash": "sanitized-value",
    "asn": 64500,
    "prefix": "sanitized-prefix"
  },
  "location_observations": [
    {"dataset": "A", "country": "US", "region": "NY", "city": "New York"},
    {"dataset": "B", "country": "US", "region": "NJ", "city": "Newark"}
  ],
  "destination": {
    "market": "US-NY",
    "accepted": true
  },
  "transport": {
    "attempts": 1,
    "exit_changes": 0,
    "success": true
  }
}

Do not overwrite the requested city with an observed city. Do not collapse disagreeing datasets into one value. Preserve the evidence that explains the disagreement.

Build the matrix around production dimensions

At minimum, cross:

every production market;
every gateway used by the application;
rotating and sticky session modes;
IPv4 and IPv6 when both are sold;
at least three separated time windows;
multiple unique exits per cell.

For a high-value market, 20–30 exits is a useful starting sample. For early provider screening, use a smaller bounded sample and expand only if the decision value justifies the traffic.

Disable retry ambiguity

Automatic retries can hide the first failing exit.

During the acceptance run:

retries = 0
concurrency = 1
redirects = explicit
connection_reuse = documented
session_id = recorded

Capture proxy connections and HTTP attempts separately. If the zero-retry test passes but production does not, investigate retry amplification and session reuse before blaming the location control.

Test rotation and stickiness separately

For rotating inventory:

create a new session identifier;
make one lookup and one authorized destination request;
close the connection;
wait for the documented rotation boundary;
repeat without reusing browser state.

For sticky inventory, retain the identifier and sample over the promised lifetime. Record the exact moment the exit changes or authentication fails.

An accurate exit that cannot remain stable still fails a sticky-session requirement.

Calculate nested accuracy

Country, region, and city matches are not independent.

country_rate = country_matches / successful_observations
region_rate  = region_matches / country_matches
city_rate    = city_matches / city_targeted_observations

Also calculate:

median and P95 distance from the requested metro center;
expected network-classification rate;
destination acceptance rate;
sticky retention rate;
unexpected exit-change rate;
repeat-exit rate;
ASN and prefix concentration.

Always report the denominator, markets, time window, address family, and match definition.

Classify disagreement patterns

Useful categories include:

one stale dataset;
administrative-boundary mismatch;
carrier aggregation with stable country but unstable city;
recent prefix reassignment;
gateway or session control defect;
IPv4/IPv6 location split.

This classification turns “the databases disagree” into a retest or escalation plan.

Make the gate executable

A machine-readable gate could look like:

country_match_min: 0.99
city_distance_p95_km_max: 50
sticky_cross_country_changes_max: 0
expected_asn_share_min: 0.90
time_windows_min: 3
result_states:
  - pass
  - conditional_pass
  - fail
  - inconclusive

The actual thresholds depend on the product and workload. The important part is defining them before reviewing results.

Security and compliance

Hash or truncate exit identifiers when full IP retention is unnecessary. Never store proxy passwords, authorization headers, cookies, or raw session tokens in the evidence bundle. Test only authorized destinations and markets, honor rate limits, and do not use location controls to bypass access restrictions or misrepresent identity.

For authorized global proxy testing, 98IP is available at https://en.98ip.com/?k=dev

Stop Rotating Proxies Until You Classify the Failure

98IP Proxy — Tue, 01 Sep 2026 15:34:21 +0000

Cloudflare’s new Adaptive Intelligence announcement highlights a reality crawler operators already feel: bot defenses change faster, and a route that succeeded yesterday may be evaluated differently today.

The dangerous response is to rotate exits immediately. That removes the evidence you need to identify the real layer of failure.

Use a failure taxonomy

Classify the event before changing transport:

Class	Typical evidence	First response
Network	timeout, reset, route-specific latency	compare one known route
Authentication	expired session, invalid credential state	refresh through the approved flow
Application	schema or page-state change	validate the parser or workflow
Rate	explicit limit, rising challenge rate	stop, back off, reduce concurrency
Policy	robots, terms, or access denial	stop and review authorization
Content	valid response but unusable data	fix quality checks, not routing

Bound the retry controller

A task should have:

exponential backoff with jitter;
a maximum attempt count;
a total elapsed-time budget;
response classes that are never retried;
a circuit breaker scoped to the target;
observability by identity, gateway, ASN, region, and session.

The core metric is:

retry_amplification = total_attempts / original_tasks

If 1,000 tasks create 3,800 attempts, the problem is not merely a 26% failure rate. The recovery controller has amplified the workload 3.8 times.

Canary before scale

For every planned route:

use a small approved URL set;
hold identity and session policy constant;
measure useful-result rate, challenges, latency, bytes, and cost;
stop on material deviation;
change one variable at a time.

Do not optimize for HTTP 200 alone. A route that returns stale, duplicate, wrong-region, or unauthorized results is not healthy.

Identity and transport are different control planes

A proxy chooses where a connection exits. It does not explain who operates the crawler or whether the target permits the use. Keep a stable, truthful identity wherever automation is allowed. Maintain an authorization register for each target. Store references to protected secrets, never credentials in telemetry.

Residential proxies can support authorized regional testing and session isolation. They are not a bypass mechanism. If a target denies the activity, rotating an exit does not create permission.

Disclosure: I work with 98IP. This post is about responsible proxy operations, not evasion. More engineering resources: https://en.98ip.com/?k=dev

Treat Crawler Identity as a Deployable Control Plane

98IP Proxy — Mon, 31 Aug 2026 06:19:51 +0000

Disclosure: I work with 98IP. This post is an engineering guide, not a claim that verification grants permission to crawl any site.

A crawler does not have one identity. It has a bundle of assertions: a user agent, an operator, an IP range, a reverse-DNS pattern, a purpose, and a contact path. When those assertions are maintained in different tickets and repositories, they drift. The result is a bot that is legitimate internally but indistinguishable from an impersonator at the edge.

The fix is to manage crawler identity like a deployable control plane.

Start with an identity manifest

Keep the public assertions in a versioned manifest owned by the same team that ships the crawler:

schema_version: 1
bot:
  name: catalog-observer
  operator: example-operator
  purpose: market-research
  contact: crawler-ops@example.invalid
network:
  user_agents:
    - "CatalogObserver/3.4"
  ip_sets:
    - production-egress-v2
  reverse_dns_suffix: crawler.example.invalid
authentication:
  web_bot_auth: required
policy:
  robots_txt: enforce
  default_rate_per_host_per_minute: 12
  stop_on_401_403_429: true

Do not put secrets in this file. It should describe claims that a destination can verify, not credentials that let someone impersonate the crawler.

Validate identity before traffic leaves

Add four checks to the release pipeline:

User-agent check: the runtime string must exactly match the manifest.
Egress check: every production address must belong to the declared set, and stale addresses must be removed.
DNS check: forward and reverse resolution must agree where reverse DNS is part of the identity contract.
Cryptographic check: if Web Bot Auth or another signed mechanism is used, verify clock skew, key rotation, and failure behavior in staging.

These tests should fail closed. A release that changes an identity surface without updating the manifest should not ship.

Observe the verifier's view

Internal health checks are insufficient. Run a small canary from each production egress region and record what an external verifier sees:

{
  "region": "eu-west",
  "observed_user_agent": "CatalogObserver/3.4",
  "forward_reverse_dns_consistent": true,
  "ip_set_version": "production-egress-v2",
  "auth_signature_valid": true,
  "policy_response": 200
}

Alert on mismatches, not merely on request failure. A 200 from one destination does not prove that the crawler is consistently identifiable elsewhere.

Separate verification from permission

Verification answers “is this traffic really from the stated operator?” Permission answers “does this site allow this activity?” They are different controls.

Before collecting data, evaluate the destination's terms, robots instructions, authentication boundaries, applicable law, and rate expectations. Stop or reduce traffic on explicit denial and on repeated 401, 403, or 429 responses. A directory listing or verified badge must never become a bypass rule.

Roll changes like network changes

Identity changes deserve staged rollout:

publish the new assertions first;
wait for DNS and directory propagation;
canary from one egress pool;
compare denial rate, authentication failures, and challenge rate;
expand only when the verifier's view matches the manifest;
retain a rollback path for the previous identity version.

The useful metrics are identity mismatch rate, verified-request ratio, policy-denial rate, stale-IP count, and time from operator change to external consistency. Raw request volume is not an identity metric.

Release checklist

[ ] Manifest reviewed by crawler and security owners
[ ] User agent and egress sets match production
[ ] Reverse DNS passes forward-confirmed checks
[ ] Signing keys rotate without overlap gaps
[ ] Contact and purpose remain accurate
[ ] Destination permission checks still run independently
[ ] Canary evidence is stored with the release
[ ] Rollback version is known

Crawler identity becomes reliable when it is testable, observable, and reversible. Treating it as a control plane reduces accidental blocking and makes impersonation easier to detect—without weakening a site owner's control.

I work with 98IP, where we build proxy infrastructure and operational guidance for compliant data workflows: https://en.98ip.com/?k=dev

AI-assisted disclosure: AI supported drafting and editing; the operational model and final review were completed for this post.