Proxy Env Vars Will Betray You: Wiring Proxies Explicitly in requests, httpx, and aiohttp

There is a class of production bug that looks like magic when you first see it. Your crawler runs perfectly in development, and in production every fetch to your own telemetry endpoint fails. Or: your "no proxy" control requests — the ones you deliberately send direct to compare against proxied results — come back with an exit IP anyway. Or my favorite: your script works for you, fails for your colleague, works again on the CI runner, and the code is byte-identical in all three places.

Nine times out of ten, the culprit is proxy configuration via environment variables: HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and their lowercase cousins. They're convenient, they're honored by default across the entire Python HTTP ecosystem, and they form an invisible global layer that silently rewires every outbound request your process makes. If you build proxy-aware crawlers, this post is about why you should treat env-var proxying as a hazard, and how to wire proxies explicitly in each of the three clients Python developers actually use.

The precedence trap

The behavior you need to internalize: in requests (and httpx, and aiohttp), when you pass no per-request or per-client proxy, the client checks trust_env (default True) and consults the environment. The effective proxy for a URL is resolved roughly in this order:

1. Explicit per-request/per-client proxy argument
2. session.trust_env-driven environment lookup (ALL_PROXY, then scheme-specific HTTPS_PROXY/HTTP_PROXY)
3. No proxy

That sounds harmless until you enumerate what else shares that environment. Your Dockerfile sets HTTPS_PROXY for the corporate gateway. Your shell profile sets it for package installs. Your orchestration platform injects it into every pod. A teammate's .env file — loaded wholesale by a twelve-factor helper — sets ALL_PROXY for the crawler's trial account. Every one of these becomes part of your crawler's network path without a single line of your code mentioning a proxy.

And NO_PROXY compounds it: exclusion is hostname-pattern matching, with library-specific wildcard semantics. NO_PROXY=*.example.com works in some clients and is treated as a literal string in others. Half the industry has a story about a NO_PROXY pattern that silently matched nothing.

Three concrete failure modes

The control-request contamination. When debugging blocks, the standard methodology is paired requests: same URL, one direct, one through the proxy exit, compare responses. If your shell has HTTPS_PROXY set, your "direct" request goes through the proxy too — and your entire differential diagnosis is comparing a proxy response with another proxy response. I've watched someone conclude a target had "no geo-differentiation at all" this way. They were measuring the same exit twice.

The telemetry sinkhole. Your crawler ships metrics and logs to an internal endpoint or a SaaS collector. Env-var proxying routes that traffic through your rotating residential pool — where it's slow, occasionally dropped, and (if the collector does IP-based rate limiting) eventually blocked. Your monitoring starts failing exactly when your crawler is struggling, which is precisely when you needed it. Worse, you're now paying proxy bandwidth prices to ship your own logs.

The subprocess surprise. Crawling stacks routinely spawn subprocesses: a headless browser, an image decoder, a pip install during cold start, a database tool in a sidecar script. All of them inherit the environment. A "temporary" export HTTPS_PROXY=... before a debugging session now proxies your Playwright's CDP traffic, your docker pulls, and everything else that process tree touches.

The fix: explicit wiring, everywhere

The discipline is simple to state: a proxy-aware crawler must never have an ambient proxy configuration. Every outbound request either goes through a proxy you named in code, or goes direct because you decided it should — and the environment has no vote.

requests

import requests

session = requests.Session()
session.trust_env = False          # 1. kill ambient env lookups
session.proxies = {"https": PROXY_URL}  # 2. declare routing in code

# Explicit direct request when you need one (e.g. control requests):
direct = requests.Session()
direct.trust_env = False           # no proxies attr => truly direct

trust_env = False also disables .netrc and other ambient auth, which in a containerized crawler is almost always what you want. Note that requests.get(url) module-level calls create a fresh session each time — set proxies explicitly per call or, better, stop using module-level calls in a crawler entirely.

httpx

httpx made a design choice worth respecting here: proxies are a client constructor argument, and there is a first-class switch:

import httpx

client = httpx.Client(
    proxy=PROXY_URL,      # applies to all requests through this client
    trust_env=False,      # ignore HTTP(S)_PROXY / ALL_PROXY / NO_PROXY
    timeout=15,
)

direct_client = httpx.Client(trust_env=False)  # guaranteed no proxy

httpx also supports mounts={} for per-scheme routing if you need http and https to behave differently. The client-scoped model composes nicely with per-domain routing: build one client per routing class (proxied-rotating, proxied-sticky, direct) and pass the right client to the right task. That's a cleaner architecture than per-request proxy arguments threaded through every call site.

aiohttp

aiohttp's trust_env defaults to True as well. The clean pattern:

import aiohttp

connector = aiohttp.TCPConnector(limit=10)
session = aiohttp.ClientSession(
    connector=connector,
    trust_env=False,          # ambient env has no vote
    proxy=PROXY_URL,          # default proxy for requests that don't override
    timeout=aiohttp.ClientTimeout(total=30),
)

# Per-request override when a specific task must go direct:
async with session.get(url, proxy=None) as resp:
    ...

One aiohttp-specific gotcha: passing proxy=None per-request disables the session default — that's your escape hatch for direct control requests, and it's documented but rarely discovered until you need it.

Making the invariant enforceable

Manual discipline decays. Two cheap mechanisms keep ambient proxies out of your crawler permanently:

Fail fast at startup. A five-line guard at process start:

import os, sys

AMBIENT = [k for k in os.environ
           if k.lower() in {"http_proxy", "https_proxy", "all_proxy"}]
if AMBIENT:
    sys.exit(
        f"Refusing to start: ambient proxy env vars found: {AMBIENT}. "
        "Unset them; this crawler configures proxies explicitly."
    )

If your deployment genuinely requires an ambient proxy (corporate egress), then whitelist it explicitly for the specific direct-path clients instead of letting it apply globally — but make that a code decision, not an inheritance.

Assert your exit in integration tests. One test that fetches a known IP-echo endpoint through each client you construct, asserting the exit IP matches what you configured (your proxy's exit range for proxied clients, your worker's real IP for direct ones). This catches regressions where a refactor silently reintroduces trust_env behavior — and it catches the control-request contamination failure mode automatically, forever.

The mental model shift

Beginners think of a proxy as a thing you add to requests. Production crawlers need the opposite framing: every request has a deliberate network path, chosen from a small routing table in code — rotating exit for bulk fetches, sticky exit for session work, direct for your own infrastructure and control comparisons. Environment variables can't express a routing table; they can only express one global answer, applied indiscriminately, invisibly, and differently on every machine your code touches.

Kill the ambient layer. Name your paths. Your future self — the one debugging a production block at 2 a.m. — will not have to wonder which of the three environments was lying.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele