Originally published on proxymint.
Proxy performance is mostly decided on the client. Tune connection reuse, concurrency and timeouts in order, then measure each phase with curl.
TL;DR
- A fresh HTTPS request through a proxy costs about four round trips before the first byte. TCP to the proxy, the CONNECT tunnel, the TLS 1.3 handshake and the request itself. A reused connection costs one.
- Connection reuse is usually the largest single win. Use one session or client per worker pool, size the pool at or above concurrency, and read and close every response body.
- Concurrency has a ceiling set by arithmetic. Requests in flight equal throughput times latency; past that, extra concurrency only lengthens the queue and raises 429s.
- Every client needs explicit timeouts. requests and Go's http.Client do not time out by default, and aiohttp waits up to 300 seconds. Set short connect, sensible read and a total deadline.
- curl's timing variables split the phases. time_connect, time_appconnect and time_starttransfer separate TCP to the proxy, tunnel plus TLS, and the target's first byte, so a slow proxy complaint names a phase.
Key takeaways
- Check num_connects before anything else. If every request opens a new connection, fix reuse before touching the provider, the protocol or the tier.
- Raise the pool defaults to match concurrency. Go keeps 2 idle connections per host and requests keeps 10 per pool unless told otherwise.
- Size concurrency from throughput and latency. 20 requests per second at 1.5 s each needs about 30 in flight; cap it in one place.
- Pick the exit near the target and keep connections open. Distance is paid on every handshake round trip, so reuse pays it once instead of four times per request.
- Compare direct and proxied curl runs over 50 samples. The median and 95th percentile gap is the proxy's share; the rest is the target and the distance.
Read the full guide on proxymint, with the commands, tables and sources.
Top comments (0)