DEV Community

Cover image for urllib proxy: ProxyHandler, auth, HTTPS tunnels and env vars
proxymint
proxymint

Posted on Originally published at proxymint.com

urllib proxy: ProxyHandler, auth, HTTPS tunnels and env vars

Originally published on proxymint.

Set a proxy in Python's urllib.request with ProxyHandler: credentials, HTTPS CONNECT tunnels, the 407 trap, HTTPS_PROXY and no_proxy, SOCKS5 and rotation.

TL;DR

  • urllib takes a proxy through ProxyHandler. urllib.request.build_opener(urllib.request.ProxyHandler({"http": PROXY, "https": PROXY})) routes both schemes, with PROXY set to http://USERNAME:PASSWORD@HOST:PORT for both keys.
  • Credentials belong in the proxy URL. ProxyHandler sends them as a Basic Proxy-Authorization header, on the CONNECT for HTTPS sites. ProxyBasicAuthHandler alone fails every https:// URL with Tunnel connection failed: 407.
  • urlopen(context=...) drops an installed proxy. It builds a fresh opener for that call, so the SSL context belongs inside the opener as an HTTPSHandler.
  • The environment can override code. On Linux a matching no_proxy entry bypasses even an explicit ProxyHandler, and urlopen reads proxy variables once, at its first call.
  • urllib has no SOCKS5 and no keep-alive. A socks5:// URL fails or is spoken to as HTTP; PySocks adds a handler. Every call opens a new connection and tunnel, which suits a rotating gateway.

Key takeaways

  • Set both the http and https keys in ProxyHandler. A handler with only an http key sends every HTTPS request direct without any error.
  • Percent-encode the username and password with quote(value, safe=""). urllib decodes them before building the header, and a raw / after an @ raises InvalidURL.
  • Pass a timeout on every call. urlopen and OpenerDirector.open wait without limit by default, so a stalled exit can hang the script.
  • Catch HTTPError before URLError. A 407 on an http:// target is an HTTPError with code 407; on an https:// target it is a URLError with the text in reason.
  • Disable proxies with ProxyHandler({}). An empty dictionary turns off environment proxies for that opener, for metadata endpoints and internal APIs.

Read the full guide on proxymint, with the commands, tables and sources.

Top comments (0)