Python requests proxy: auth, sessions, SOCKS5 and rotation

Originally published on proxymint.

Set a proxy in Python requests the right way: the proxies dict and its https key, auth encoding, sessions, SOCKS5, rotation, retries and the silent bypass.

TL;DR

• The proxies dict keys name the target, not the proxy. requests picks the http or https entry by the scheme of the URL being fetched. A dict with only an http key sends every HTTPS page from your own IP, with no error.
• Both values should be http:// URLs. For HTTPS sites requests sends CONNECT through an http:// proxy and TLS runs end to end with the site. An https:// proxy URL means TLS to the proxy itself, a different setup that urllib3 added in 1.26.0.
• A 407 on an HTTPS page raises ProxyError. The refusal answers the CONNECT, so no response object exists and a status_code check never fires. Only plain http:// pages return a 407 response.
• Sessions reuse connections, and that can pin the exit. If a gateway picks the exit when a connection opens, every call on one kept-alive tunnel leaves from the same address. A short test shows what a given gateway does.
• Environment proxies overwrite session.proxies. HTTP_PROXY and HTTPS_PROXY on a CI runner can silently reroute traffic. Setting trust_env to False on the session makes the dict the only source.

Key takeaways

• Set both keys and a timeout on every call. Use {"http": PROXY, "https": PROXY} with an http:// proxy URL and timeout=(5, 30), because requests never times out by default.
• Percent-encode credentials and load them from the environment. Run username and password through urllib.parse.quote with safe set to an empty string, and pin requests 2.31.0 or later.
• Catch ProxyError, not status 407, for HTTPS targets. Log the exception text, and treat a 407 with known-good credentials as an encoding or package problem.
• Assert the exit IP before the run and on a sample during it. Compare against the direct IP, and compare one shared session with fresh sessions to see whether rotation follows connections.
• Retry only 429 and 5xx, with backoff. Never retry 407 or 403, and budget retries on a per-GB plan, because each one is a full download.

Read the full guide on proxymint, with the commands, tables and sources.