Shave the Bytes: Traffic Engineering for GB-Billed Proxy Traffic

Residential proxy traffic is billed by the gigabyte, and most teams treat that number as fixed: the pages are as big as they are, the vendor charges what it charges, and the only lever is haggling over the rate card or switching providers.

That's leaving the biggest lever untouched. On a GB-billed proxy, every byte you don't transfer is pure margin, and the difference between a naive fetch and an engineered one is routinely 5–10x on the same workload. A pipeline that pulls 800 GB a month can often do identical work in 100–150 GB — a savings no negotiation will ever match.

This post is a checklist of byte-shaving techniques, in descending order of impact, with the Python to implement each. Everything here is ordinary HTTP engineering; the proxy angle just makes it worth real money.

1. Response compression is not automatic through a proxy

The single most common waste: clients that don't negotiate compression. Browsers always send Accept-Encoding: gzip, br; naive Python scrapers often don't, and servers respond with uncompressed HTML — 4–8x larger than the gzipped equivalent. Worse, some clients do send the header but the proxy middleware strips or mishandles it, and nobody notices because nobody counts bytes.

import requests

s = requests.Session()
s.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
    "Accept-Encoding": "gzip, deflate",  # requests + urllib3 handles decompress
})
s.proxies = {"http": "http://user:pass.gate.thordata.com:7000",
             "https": "http://user:pass.gate.thordata.com:7000"}
r = s.get("https://target.example/products/123")
print(len(r.content), "bytes received")   # wire size before decompression

Measure the wire size, not len(r.text) — check r.raw or the Content-Length of the actual response. urllib3 decompresses transparently, so the savings are invisible unless you look at what crossed the proxy.

2. Never fetch HTML to read JSON

Most modern sites render from a JSON API, and the HTML page is a 300 KB shell wrapping data that's also available as a 20 KB XHR response. If you're scraping product data, find the internal API the page's own JavaScript calls (DevTools → Network → XHR) and call it directly. This is a 10–15x byte reduction for the same data, with the added reliability of structured payloads over DOM scraping.

The usual objection — "internal APIs can change" — is real, but the HTML layout changes too, and more often. Track both; fetch the JSON.

3. Trim the request side, not just the response

Requests are metered too. Three habits matter:

• Don't ship fat cookie jars. A requests.Session accumulates cookies per domain; some auth flows leave multi-KB jars that get re-sent on every request. Periodically rebuild sessions, or use a cookiejar policy that drops bulky tracking cookies you don't need.
• Drop the boilerplate headers you copied. A realistic browser header set is ~1–2 KB. You need enough to look coherent, but a 40-header monster from a "browser headers" GitHub list is unnecessary ballast on every request.
• Paginate with precision. If the API accepts fields= or limit= parameters, ask for only the fields you store. Fetching full product records to extract two fields wastes the other 90% of every response.

4. Conditional requests: let the server say "unchanged"

For monitoring workloads — price checks, stock checks, rank checks — most fetches return data you already have. HTTP gives you the mechanism to pay almost nothing for that answer:

• ETag / If-None-Match: server answers 304 Not Modified with an empty body.
• Last-Modified / If-Modified-Since: same idea, timestamp-based.

class ConditionalFetcher:
    """Persistent ETag cache — 304 responses cost ~200 bytes, not 300 KB."""
    def __init__(self, session):
        self.s = session
        self.etags: dict[str, str] = {}

    def get(self, url):
        headers = {}
        if url in self.etags:
            headers["If-None-Match"] = self.etags[url]
        r = self.s.get(url, headers=headers, timeout=20)
        if r.status_code == 304:
            return None          # unchanged; caller keeps its stored copy
        if etag := r.headers.get("ETag"):
            self.etags[url] = etag
        return r

fetcher = ConditionalFetcher(s)
result = fetcher.get("https://target.example/products/123")
if result is None:
    ...  # no change since last poll; total cost: ~0.2 KB

Support varies — dynamic pages often omit validators, and some CDNs strip them — but where it works, an unchanged page costs a few hundred bytes instead of a few hundred kilobytes. On a monitor that polls 10,000 URLs hourly where 5% change per poll, this is the difference between a staggering bill and a rounding error.

For APIs without validators, implement the same logic yourself: fetch a cheap signature (a updated_at field from a list endpoint) first, and only fetch the full record when the signature moves. Two-stage fetch, same economics.

5. Block the asset firehose (if you must render)

When a page genuinely requires JavaScript rendering, the browser will happily pull megabytes of images, fonts, and analytics scripts through your metered proxy. Intercept and abort them:

from playwright.sync_api import sync_playwright

BLOCK_TYPES = {"image", "font", "media"}
BLOCK_URLS = ("googletagmanager.com", "google-analytics.com",
              "doubleclick.net", "facebook.net", "hotjar.com")

def render(url, proxy_url):
    with sync_playwright() as p:
        browser = p.chromium.launch(proxy={"server": proxy_url})
        ctx = browser.new_context()
        ctx.route("**/*", lambda route: (
            route.abort()
            if route.request.resource_type in BLOCK_TYPES
            or any(d in route.request.url for d in BLOCK_URLS)
            else route.continue_()
        ))
        page = ctx.new_page()
        page.goto(url, wait_until="domcontentloaded")
        html = page.content()
        browser.close()
        return html

Combined with wait_until="domcontentloaded" instead of "load", blocking images/fonts/trackers typically cuts rendered-page traffic 60–80%. You don't need the product photos — you need the price next to them.

Measure the wire, not the vibes

All of this is invisible unless you account bytes explicitly. Wrap the session:

class MeteredSession(requests.Session):
    def __init__(self, *a, **kw):
        super().__init__(*a, **kw)
        self.wire_bytes = 0

    def send(self, request, **kw):
        self.wire_bytes += len(request.body or b"") + len(str(request.headers))
        resp = super().send(request, **kw)
        # raw wire length when available; header estimate otherwise
        self.wire_bytes += int(resp.headers.get("Content-Length",
                              len(resp.raw.read()) if hasattr(resp.raw, "read") else 0))
        return resp

Log wire_bytes per domain per day, multiply by your per-GB rate, and you have the only dashboard that matters: dollars per successful row, trending over time. Every technique above should show up as a visible drop on that chart — and the chart tells you which pages still leak.

The stack-up

On a real price-monitoring workload (5,000 SKUs, hourly polls), a measured before/after:

Stage	Monthly proxy traffic
Naive: HTML pages, no compression	~780 GB
+ gzip, API endpoints instead of HTML	~210 GB
+ conditional/two-stage fetch	~45 GB
+ trimmed headers and cookie hygiene	~38 GB

Same data in the warehouse. Twenty times less traffic. At residential rates, that's a car-payment-sized monthly difference on a single pipeline — and unlike vendor switching, none of it risks new block profiles or migration bugs.

The next time the proxy bill triggers a meeting, bring the wire-bytes chart instead of the rate card.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele