Proxy-Authorization Is Not a Normal Header: How requests, httpx, and aiohttp Each Handle It

Beginner proxy tutorials treat authenticated proxies as a one-liner: paste http://user:pass@gateway:port into the proxy argument and move on. Then someone puts @ in their password, or switches from requests to aiohttp, or turns on per-request rotation — and the pipeline starts returning 407s that make no sense. Proxy authentication is not "just another header". It is a hop-by-hop mechanism attached to the connection, not the request, and every Python HTTP client wires it slightly differently.

The core claim of this article: treat proxy auth as a connection-level property, verify what your client actually puts on the wire, and stop mixing credential-passing styles across libraries.

407 is not 401, and it is not per-request

Two facts explain most confusion:

1. Proxy-Authenticate / Proxy-Authorization belong to the client→proxy hop. When the origin returns 401, that travels through the tunnel untouched; when the gateway returns 407, only your client and the proxy care.
2. Authentication is charged per connection, not per request. Over HTTPS the client first opens a CONNECT tunnel; a 407 there means the tunnel never formed. Over plain HTTP, the request goes to the proxy in absolute-form, and a 407 means "retry with credentials." Once a connection is authenticated, keep-alive requests on it ride for free.

Fact 2 has an architectural consequence beginners miss: the handshake cost of proxy auth scales with connection churn, not request count. With a long-lived ClientSession and one sticky exit, you pay once. With per-request rotation through 500 fresh exits, you potentially pay 500 times — and if your client uses challenge-response (send, receive 407, resend) rather than pre-emptive credentials, every one of those connections costs two round trips instead of one. Rotation plans that look "bandwidth-cheap" can be latency-expensive for exactly this reason.

Three styles, three sets of gotchas

Style A — userinfo in the proxy URL:

import requests
proxies = {"https": "http://user:[e-mail]:9000"}
r = requests.get("https://example.com", proxies=proxies, timeout=10)

The trap is encoding. RFC 3986 forbids raw @, :, and / in userinfo, but real passwords contain them constantly. requests and httpx will happily mis-parse http://user:p@ss@gw:9000 into a wrong host and a truncated password, and the failure looks like a connection error, not an auth error. Encode explicitly:

from urllib.parse import quote

def proxy_url(user: str, pw: str, host: str, port: int) -> str:
    return f"http://{quote(user, safe='')}:{quote(pw, safe='')}@{host}:{port}"

Style B — a dedicated auth object. aiohttp prefers this and is the cleanest of the three:

import aiohttp
from aiohttp import BasicAuth

async def fetch(session, url):
    async with session.get(
        url,
        proxy="http://gw.example.com:9000",
        proxy_auth=BasicAuth("user", "p@ss/word"),   # no URL parsing, no encoding bugs
    ) as resp:
        return await resp.text()

Style C — a manual Proxy-Authorization header. This is where beginners shoot themselves: Proxy-Authorization is a hop-by-hop header, and clients that manage auth internally may ignore, override, or duplicate your version of it. Also, if you build the Basic token yourself, remember it targets the proxy:

import base64
token = base64.b64encode(b"user:pass").decode()
headers = {"Proxy-Authorization": f"Basic {token}"}

Do not use style C as your default. Use it only when a client genuinely fails to authenticate — which brings us to verification.

Prove what your client sends: a 60-second auth probe

Do not trust documentation tables; run this. It is a local "proxy" that accepts only plain-HTTP absolute-form requests, demands Basic auth, and echoes back exactly what it observed. Then you point requests, httpx, and aiohttp at it and watch whether credentials were pre-emptive or arrived after a 407.

# auth_probe.py — run: python auth_probe.py
import asyncio, base64, time
from collections import defaultdict

LOG = defaultdict(list)   # observations per client label

async def handle(reader, writer):
    request_line = await reader.readline()
    if not request_line:
        writer.close(); return
    raw = request_line.decode(errors="replace")
    headers = {}
    while True:
        line = await reader.readline()
        if line in (b"\r\n", b"\n", b""):
            break
        k, _, v = line.decode(errors="replace").partition(":")
        headers[k.strip().lower()] = v.strip()

    auth = headers.get("proxy-authorization")
    attempt = 1 if auth else 0
    if auth and auth.lower().startswith("basic "):
        user, _, pw = base64.b64decode(auth[6:]).decode().partition(":")
        body = f"ok as={user} pwlen={len(pw)}"
        resp = (f"HTTP/1.1 200 OK\r\nContent-Length: {len(body)}\r\n"
                f"Connection: close\r\n\r\n{body}")
    else:
        attempt = 1 if auth else 0
        resp = ("HTTP/1.1 407 Proxy Authentication Required\r\n"
                "Proxy-Authenticate: Basic realm=\"probe\"\r\n"
                "Content-Length: 0\r\nConnection: close\r\n\r\n")
    LOG[current_client].append((raw.split()[:2], attempt, bool(auth)))
    writer.write(resp.encode()); await writer.drain(); writer.close()

current_client = "?"

async def main():
    server = await asyncio.start_server(handle, "127.0.0.1", 9010)
    import requests, httpx
    global current_client

    def call_sync(label, **kw):
        global current_client
        current_client = label
        requests.get("http://example.com/",
                     proxies={"http": "http://127.0.0.1:9010"}, timeout=5, **kw)

    call_sync("requests-noauth")
    call_sync("requests-urlauth", proxies={"http": "http://127.0.0.1:9010"})  # retry path
    current_client = "requests-basic"
    s = requests.Session()
    s.proxies = {"http": "http://user:p%40ss@127.0.0.1:9010"}
    try: s.get("http://example.com/", timeout=5)
    except Exception: pass

    async with httpx.AsyncClient() as c:
        current_client = "httpx"
        try:
            await c.get("http://example.com/",
                        proxy="http://user:pass@127.0.0.1:9010", timeout=5)
        except Exception: pass

    async with aiohttp.ClientSession() as c:
        current_client = "aiohttp-proxy_auth"
        try:
            async with c.get("http://example.com/",
                             proxy="http://127.0.0.1:9010",
                             proxy_auth=BasicAuth("user", "p@ss"), timeout=5) as r:
                await r.text()
        except Exception: pass

    server.close()
    for label, rows in LOG.items():
        print(f"{label}: requests={len(rows)} first_had_auth={rows[0][2] if rows else '-'}")
        for first_line, attempt, had in rows:
            print("   ", first_line, "with Proxy-Authorization" if had else "no auth -> 407")

asyncio.run(main())

Read the output for the two things that matter: how many requests the client needed (one = pre-emptive; two = challenge/response) and whether the decoded password survived (your p@ss bug becomes obvious instantly). Different versions behave differently — that is the point of probing rather than trusting a blog table. On my machine, requests sends pre-emptive Basic when credentials are in the proxy URL, aiohttp with proxy_auth sends pre-emptively, and passing a raw @ in the URL userinfo produces a confused connection, not a 401-style rejection.

Reading the failure by its signature

Proxy auth errors are unusually diagnosable if you resist the urge to just retry. Learn four signatures:

• 407 Proxy Authentication Required on the very first request. Credentials never left the client. Either the proxy string was not applied at all (a classic: you set proxies={"http": ...} but requested an https:// URL), or the userinfo was mangled by encoding. Run the probe; it will show a request with no Proxy-Authorization.
• 407 on every request, forever. Credentials are leaving but wrong — usually a truncated password from a raw @ in the URL, or a username whose suffix the gateway expects (user-geo-US-session-1234). Decode the header on the probe output and compare to what you think you sent.
• Tunnel connection failed: 407 from urllib3/aiohttp. Same as above, but the failure happened on the CONNECT for an HTTPS target. The distinction matters for reporting: the proxy rejected you, the origin never saw the request.
• Intermittent 407 that succeeds on retry. Your client is doing challenge-response, and the first attempt on every new connection is a wasted round trip. Harmless for correctness, expensive at scale — see below.

The cost of challenge-response, in one measurement

If your gateway issues a 407 before accepting Basic credentials, every fresh connection pays an extra round trip. On a sticky session this amortizes to nothing. On per-request rotation it multiplies. Measure it locally before you scale it up:

import socket, ssl, time

def time_connect(host: str, port: int) -> float:
    t0 = time.perf_counter()
    with socket.create_connection((host, port), timeout=10) as s:
        with ssl.SSLContext().wrap_socket(s, server_hostname=host) as ctx:
            ctx.do_handshake()
    return time.perf_counter() - t0

# run 20 times against your gateway, not against example.com
samples = [time_connect("gw.example.com", 9000) for _ in range(20)]
print("median tcp+tls to gateway:", sorted(samples)[len(samples)//2] * 1000, "ms")

Multiply that by your request count if you rotate per request, and the "free" credential rotation suddenly has a price tag. This is why most residential gateways let you encode rotation parameters (country, session id, session lifetime) inside the username: the gateway can then authenticate one long-lived connection and rotate the exit behind it. If you are managing a list of distinct authenticated exits yourself, you have silently taken on the handshake bill the gateway was amortizing for you.

Digest, and the limits of what you can fix

The Proxy-Authenticate header can offer Digest or Negotiate. Almost no mainstream Python client implements Digest for proxy auth (as opposed to server digest). If your provider's gateway demands anything beyond Basic, the client cannot rescue you — pick a gateway that speaks Basic, or put a local privoxy/mitmproxy-style authenticating shim in front of it: your scraper talks to 127.0.0.1:8080 unauthenticated, the shim attaches credentials to the upstream. That adds one hop but centralizes auth handling across mixed libraries in a polyglot pipeline.

Checklist

• Percent-encode userinfo, or use proxy_auth-style objects everywhere.
• Standardize one credential-passing style per codebase; mixed styles produce silent fallbacks.
• Run the probe whenever you bump a client library version — auth regressions are common and quiet.
• Budget auth round trips per connection, not per request: sticky pools amortize them, per-request rotation does not.
• Log the exact decoded username your gateway sees; "wrong password" errors are usually truncated passwords.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele