Every Layer of the PyPI Scraping Stack Treats Proxies Differently: Building One Compatibility Layer

Pick any three packages from the PyPI scraping stack — say requests, httpx, aiohttp, maybe httpretty for testing, plus playwright for the stubborn pages — and try to route them all through the same proxy configuration. You will discover, usually at the worst possible moment, that each layer has its own opinion about what a proxy is: an environment variable it may or may not honor, a dict with American-or-British spelling, a per-request keyword, or a string that must not contain a schema because it will be parsed differently than the one in the other library.

None of this is a bug in any single package. It is the predictable result of the ecosystem growing up in parallel. But for anyone building a collection pipeline, it means the proxy layer — the part you most need to be uniform, testable, and swappable — is scattered across incompatible conventions.

My argument: in a real scraping stack, proxy configuration deserves one compatibility layer of your own — a single module that owns the provider credentials and the session conventions, and emits correctly-shaped proxy objects for every consumer. You should never write a proxy URL string in business code again.

A tour of the incompatibilities

Let me make the claim concrete. Here is the same logical configuration — "route through exit X, sticky session Y" — expressed in four layers of the standard stack:

requests wants a dict, per-session or per-request:

import requests

proxies = {"http": proxy_url, "https": proxy_url}
r = requests.get(url, proxies=proxies)

It also honors HTTP_PROXY/HTTPS_PROXY env vars — which sounds convenient until a CI runner leaks a corporate proxy into your container and your residential traffic silently routes through your employer's egress, acquiring a datacenter fingerprint you never asked for.

`httpx also takes a dict, but the per-request form is proxy= (singular), and in async mode proxies are bound at the AsyncClient` level — creating a client per exit is the only way to get pool isolation.

aiohttp takes a plain string in proxy=, per request, and pointedly does not read environment variables (a deliberate design choice that surprises people coming from requests). Its TrustEnv flag flips that.

playwright wants the proxy at browser-launch time, as a separate proxy={"server": ..., "username": ..., "password": ...} object — parsed, not a URL string — and it applies to the whole browser context, which is exactly why one-context-per-exit is the standard pattern for browser fleets.

Same intent, four dialects. Now multiply by the places proxies get configured — code, env, .env files, Docker compose, orchestrator secrets — and by the session/stickiness convention each provider embeds in the username. The matrix is why "just set the env var" pipelines behave differently in staging and production.

The compatibility layer

The fix is one module with a narrow contract: it knows your provider and your session semantics, and it can produce the right shape for each consumer. Here is a version I have used in production, trimmed to its load-bearing parts:

# proxy_compat.py
from __future__ import annotations

import os
from dataclasses import dataclass
from typing import Any


@dataclass(frozen=True)
class ProxyConfig:
    """The single source of truth. Everything else is a projection."""
    scheme: str = "http"
    host: str = "gate.thordata.com"
    port: int = 7000
    username: str = ""
    password: str = ""

    def with_session(self, session_id: str) -> "ProxyConfig":
        # Provider convention: sticky sessions via username suffix.
        # This is the ONLY place in the codebase that knows this.
        return ProxyConfig(
            self.scheme, self.host, self.port,
            f"{self.username}-session-{session_id}",
            self.password,
        )

    @property
    def url(self) -> str:
        auth = f"{self.username}:{self.password}@" if self.username else ""
        return f"{self.scheme}://{auth}{self.host}:{self.port}"

    # --- projections for each consumer ---

    def for_requests(self) -> dict[str, str]:
        return {"http": self.url, "https": self.url}

    def for_httpx(self) -> str:
        return self.url

    def for_aiohttp(self) -> str:
        return self.url

    def for_playwright(self) -> dict[str, str]:
        return {
            "server": f"{self.scheme}://{self.host}:{self.port}",
            "username": self.username,
            "password": self.password,
        }

    @classmethod
    def from_env(cls) -> "ProxyConfig":
        # Explicit beats ambient: read OUR variables, not HTTP_PROXY.
        return cls(
            host=os.environ["PROXY_HOST"],
            port=int(os.environ.get("PROXY_PORT", "7000")),
            username=os.environ["PROXY_USER"],
            password=os.environ["PROXY_PASS"],
        )

Notice the deliberate asymmetry: from_env reads your variables (PROXY_HOST, PROXY_USER) and ignores the ambient HTTP_PROXY family entirely. This is the most important line in the file. Ambient proxy configuration is how residential traffic ends up double-hopped through a corporate egress — same exit IP for every request, datacenter ASN, TLS fingerprint of a Go middleware appliance. Your carefully selected pool never sees a single byte.

Wiring it into each layer

With the projection methods in place, the consumers become one-liners that never touch a URL string:

import requests
import httpx
import aiohttp
from playwright.async_api import async_playwright
from proxy_compat import ProxyConfig

cfg = ProxyConfig.from_env()
sticky = cfg.with_session("sku-B08N5WRWNW")

# requests — per-request, no ambient surprises
page = requests.get("https://example.com/dp/B08N5WRWNW",
                    proxies=sticky.for_requests(), timeout=30)

# httpx — client-per-exit for pool isolation
with httpx.Client(proxy=sticky.for_httpx(), timeout=30) as client:
    page = client.get("https://example.com/dp/B08N5WRWNW").text

# aiohttp — per-request proxy string, TrustEnv off
async def fetch(url: str) -> str:
    async with aiohttp.ClientSession(trust_env=False) as s:
        async with s.get(url, proxy=sticky.for_aiohttp()) as r:
            return await r.text()

# playwright — proxy at context launch
async def render(url: str) -> str:
    async with async_playwright() as pw:
        browser = await pw.chromium.launch(proxy=sticky.for_playwright())
        ctx = await browser.new_context()
        page = await ctx.new_page()
        await page.goto(url)
        html = await page.content()
        await browser.close()
        return html

Every consumer now expresses intent ("this fetch goes through the sticky exit for this SKU") and nothing about mechanics.

Testing the layer without burning traffic

A compatibility layer you can't unit-test is a compatibility layer you'll fear. The trick is to run the projections against a local stub proxy and assert the shapes:

# test_proxy_compat.py
import pytest
from proxy_compat import ProxyConfig

def test_url_roundtrip():
    cfg = ProxyConfig(username="u", password="p")
    assert cfg.url == "http://u:[e-mail]:7000"

def test_session_suffix_isolated_to_username():
    base = ProxyConfig(username="u", password="p")
    s = base.with_session("abc")
    assert s.username == "u-session-abc"
    assert s.password == "p"            # password untouched
    assert base.username == "u"         # caller's config unmutated (frozen)

def test_playwright_shape_has_no_embedded_auth():
    cfg = ProxyConfig(username="u", password="p")
    pw = cfg.for_playwright()
    assert "u:p@" not in pw["server"]   # auth is separate keys
    assert set(pw) == {"server", "username", "password"}

These tests cost nothing to run and catch the class of bug that otherwise surfaces as a 407 from a provider at 2 a.m. — a misplaced suffix, a mutated shared config, auth leaking into a field a consumer parses differently.

One more wrinkle: SOCKS and authentication schemes

Two further incompatibilities surface once you leave plain HTTP gateways behind. SOCKS5 proxies are supported natively by aiohttp (with python-socks installed) and by httpx, but requests needs the third-party requests[socks] extra — and Playwright's Chromium build ignores SOCKS authentication entirely, which silently fails on providers that require it. If your provider offers both HTTP and SOCKS gateways, the compatibility layer is also where that capability flag belongs: expose supports_socks on the config, and let consumers that genuinely need UDP-adjacent behavior opt in, while everything else stays on the HTTP gateway where the whole stack behaves identically. The rule of thumb I use: every capability difference between consumers becomes a field on ProxyConfig, never an if statement at the call site.

The provider portability dividend

The deepest payoff shows up the day you evaluate a second provider. Because every convention — gateway hostname, session suffix grammar, port-per-geo-country schemes — lives behind ProxyConfig, a provider bake-off becomes: subclass, override with_session, run the same test suite against the new gateway, and measure success rates per tier with the rest of the stack untouched. I've done this switch in an afternoon. The teams that embed provider URL formats in forty scripts do it in a quarter, or not at all, which is how you end up paying for a pool you stopped trusting years ago.

One compatibility layer, five projections, zero proxy strings in business code. It is a small discipline with an outsized return — the stack above it finally behaves like one system instead of four.

---

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele