DEV Community

Cover image for aiohttp proxy: auth, trust_env, SOCKS5, limits and rotation
proxymint
proxymint

Posted on Originally published at proxymint.com

aiohttp proxy: auth, trust_env, SOCKS5, limits and rotation

Originally published on proxymint.

Set a proxy in Python aiohttp: proxy= per session or request, auth after the 3.14 proxy_auth deprecation, trust_env, SOCKS5, connector limits and each error.

TL;DR

  • Pass the proxy as proxy= and keep http:// even for HTTPS sites. ClientSession(proxy="http://USERNAME:PASSWORD@HOST:PORT") covers both, a proxy= on one get() overrides it, and the session form needs aiohttp 3.11 or later, so run pip show aiohttp before copying an old tutorial.
  • If your password has a /, # or ? in it, percent-encode it, or you get InvalidURL with the password printed in the message. Skip proxy_auth=BasicAuth(...) too: it warns on every call since 3.14, and its proxy_headers replacement works per request only and does nothing for plain-HTTP targets.
  • HTTPS_PROXY does nothing until you pass trust_env=True, and even then ALL_PROXY is skipped and an https:// value in HTTPS_PROXY is dropped with a log warning, so requests go direct without an error. An explicit proxy= saves you that hunt.
  • On an HTTPS site a 407 arrives as ClientHttpProxyError, not a status code, and because it subclasses ClientResponseError you catch it first; only plain http:// targets give you resp.status == 407.
  • For SOCKS5 you need aiohttp-socks, which rejects socks5h://. ProxyConnector.from_url("socks5://...") already sends the hostname to the proxy by default; rdns=False is what turns that off.

Key takeaways

  • Standardise on one proxy URL with quote(..., safe="") credentials. It works for HTTPS and plain-HTTP targets, on the session or per request, with no DeprecationWarning on 3.14.5.
  • Cap tunnels with TCPConnector(limit=N) and a semaphore of the same size. limit_per_host counts target hosts, not the proxy port, and requests queued for the pool burn their total timeout.
  • Set ClientTimeout yourself. The default is 300 seconds total, so a dead exit holds a slot for five minutes before anything fails.
  • Run the reused-vs-fresh exit test before trusting rotation. Keep-alive reuses one tunnel for 15 seconds, and force_close=True is the switch for a new tunnel per request.
  • Retry 429, 5xx and timeouts in the job, never 407 or 403. aiohttp has no built-in retry, and on per-GB plans every retry is extra traffic.

Read the full guide on proxymint, with the commands, tables and sources.

Top comments (0)