When a proxy request fails, “retry it” is an attractive default. It is also how a small authentication mistake or temporary rate limit turns into a noisy incident.
I work with the 98IP team. This playbook is the troubleshooting sequence I recommend before increasing concurrency, rotating endpoints, or widening a proxy pool.
Start with the layer, not the error string
A useful request model has five layers:
- client and connection pool;
- proxy authentication;
- proxy egress and routing;
- target service;
- application parsing and business logic.
Hold the target, region, session mode, and concurrency constant. Capture a request ID, UTC timestamp, HTTP status, relevant headers, connection phase, and duration. Change one variable per experiment.
407 is not a rotation problem
407 Proxy Authentication Required usually means the proxy needs valid credentials. Check the endpoint, port, username, password, authentication method, and IP allowlist. Make sure URL encoding has not changed the credential string and that Authorization for the target is not being confused with Proxy-Authorization.
Run one request at low concurrency. If every endpoint fails, investigate account state or permissions. If only one region fails, isolate that region. Rotating faster cannot fix invalid authentication.
429 needs less pressure
429 Too Many Requests is a rate-limit signal. It may come from the target, proxy, or another intermediary. Inspect Retry-After and Proxy-Status when available.
The correct response is bounded backoff:
delay = min(cap, base * 2^attempt) + random_jitter
Also cap attempts, enforce a total time budget, and keep a per-domain concurrency limit. Once the budget is exhausted, move the task to a delayed queue. Switching IPs without reducing pressure is not a rate-limit strategy.
“Timeout” is five different measurements
Measure DNS, TCP connect, TLS handshake, time to first byte, and response-body read separately.
- Connect timeout: routing, firewall, or node availability.
- TLS timeout: protocol negotiation or certificate path.
- First-byte timeout: slow target, congested egress, or expensive request.
- Read timeout: large payload, bandwidth pressure, or streaming behavior.
Use separate connect and read timeouts. A single global timeout hides the phase that matters.
Connection reset needs a control test
A reset can originate at the proxy, target, intermediate device, or local pool. Disable connection reuse for a control run. Compare TLS versions, HTTP/2 or HTTP/3 negotiation, idle lifetime, request-body size, and requests per connection.
If resets cluster on one region or node, quarantine it. If they appear only at high concurrency, reduce requests per connection and shorten the queue.
A retry matrix that does not create a storm
| Failure | Default action | Automatic retry? |
|---|---|---|
| 407 | Repair credentials or allowlist | No |
| 429 | Honor waiting instructions; back off | Yes, strictly bounded |
| Connect timeout | Try a healthy node | Yes, strictly bounded |
| Read timeout | Inspect payload and target latency | Conditional |
| Connection reset | Pause; open a fresh connection | Conditional |
| Deterministic 4xx | Change request conditions | No |
Minimum evidence for every failure
Keep the request ID, time, target hostname, proxy region, session type, status, failure phase, duration, retry count, and final outcome. Do not log complete proxy passwords, cookies, personal data, or sensitive response bodies.
Operational checklist
- Does one request succeed before concurrency is increased?
- Are 407 failures separated from target-side 401 responses?
- Does 429 logic honor server guidance and cap retries?
- Are timing phases measured separately?
- Do resets correlate with region, node, protocol, reuse, or concurrency?
- Is there a maximum attempt count, time budget, and circuit breaker?
- Are collection scope and data retention documented?
Reliable proxy operations come from controlled experiments and bounded recovery, not from treating every exception as permission to rotate faster.
If you are evaluating regional and session behavior for an authorized workflow, the 98IP team maintains an overview of available proxy options at https://en.98ip.com/?k=dev.
Use proxy infrastructure only where you have permission, respect target terms and rate limits, and minimize the personal data you collect or retain.
Tags: webscraping, proxy, performance, tutorial
Top comments (0)