Originally published on proxymint.
Set a proxy in Python aiohttp: proxy= per session or request, auth after the 3.14 proxy_auth deprecation, trust_env, SOCKS5, connector limits and each error.
TL;DR
- Pass the proxy as proxy= and keep http:// even for HTTPS sites. ClientSession(proxy="http://USERNAME:PASSWORD@HOST:PORT") covers both, a proxy= on one get() overrides it, and the session form needs aiohttp 3.11 or later, so run pip show aiohttp before copying an old tutorial.
- If your password has a /, # or ? in it, percent-encode it, or you get InvalidURL with the password printed in the message. Skip proxy_auth=BasicAuth(...) too: it warns on every call since 3.14, and its proxy_headers replacement works per request only and does nothing for plain-HTTP targets.
- HTTPS_PROXY does nothing until you pass trust_env=True, and even then ALL_PROXY is skipped and an https:// value in HTTPS_PROXY is dropped with a log warning, so requests go direct without an error. An explicit proxy= saves you that hunt.
- On an HTTPS site a 407 arrives as ClientHttpProxyError, not a status code, and because it subclasses ClientResponseError you catch it first; only plain http:// targets give you resp.status == 407.
- For SOCKS5 you need aiohttp-socks, which rejects socks5h://. ProxyConnector.from_url("socks5://...") already sends the hostname to the proxy by default; rdns=False is what turns that off.
Key takeaways
- Standardise on one proxy URL with quote(..., safe="") credentials. It works for HTTPS and plain-HTTP targets, on the session or per request, with no DeprecationWarning on 3.14.5.
- Cap tunnels with TCPConnector(limit=N) and a semaphore of the same size. limit_per_host counts target hosts, not the proxy port, and requests queued for the pool burn their total timeout.
- Set ClientTimeout yourself. The default is 300 seconds total, so a dead exit holds a slot for five minutes before anything fails.
- Run the reused-vs-fresh exit test before trusting rotation. Keep-alive reuses one tunnel for 15 seconds, and force_close=True is the switch for a new tunnel per request.
- Retry 429, 5xx and timeouts in the job, never 407 or 403. aiohttp has no built-in retry, and on per-GB plans every retry is extra traffic.
Read the full guide on proxymint, with the commands, tables and sources.
Top comments (0)