Cost Optimization for Proxy Traffic: Cutting Your Per-GB Scraping Bill

Ask anyone running a scraping pipeline at scale what their biggest line item is, and it's usually not the servers or the engineering time — it's bandwidth. Residential and mobile proxy traffic is billed per gigabyte, and a careless collector can easily push 10–50 GB a day fetching pages whose useful payload is maybe 3% of the bytes transferred. When your proxy bill is $5/GB, the difference between a lean pipeline and a wasteful one is the difference between hundreds and thousands of dollars a month.

The good news: this is one of the highest-leverage optimization problems in scraping, because the waste is structural and therefore fixable. Here's the playbook, roughly in order of ROI.

First, Measure Where the Bytes Go

You can't optimize what you don't meter. Log bytes per request at the proxy layer and break it down by endpoint type. The typical profile, in my experience, looks like:

• 40–60% of bytes: images, fonts, media (almost never needed for scraping)
• 15–25%: JavaScript bundles (needed only if you're rendering)
• 10–20%: HTML document itself
• 5–15%: JSON/API responses (the actual payload)
• 5–10%: tracking/analytics calls

Which means for a request-and-parse pipeline (no JS rendering), 60–80% of a naive page's bytes are waste — if you're even fetching them. The first question is even more basic.

Optimization 1: Don't Fetch the Page — Fetch the API

The single biggest win in scraping cost is discovering that most pages are cosmetics over JSON. Modern sites hydrate from internal or public API endpoints, and the JSON is 5–20x smaller than the rendered page, requires no browser, and is more stable to parse. Before scraping any site at volume: open devtools, watch the network tab during normal browsing, and look for the XHR calls that carry the actual data. Product listings, prices, reviews, availability — they're almost always in JSON payloads.

When an API exists, the decision tree is: use it (through the same geo/session setup you'd use for pages). When it doesn't, scrape the HTML directly (requests + parser, no rendering) — which, again, skips all the asset bytes. Reserve headless browsers for the genuinely JS-only minority.

Optimization 2: Block What You Don't Need (for the Rendering Minority)

When you must render, block the asset classes you don't consume. In Playwright:

from playwright.sync_api import sync_playwright

BLOCK_TYPES = {"image", "font", "media"}
BLOCK_URL_PATTERNS = (
    "google-analytics.com", "googletagmanager.com", "doubleclick.net",
    "facebook.net", "hotjar.com", "segment.io", "fullstory.com",
)

def lean_page(route, request):
    if request.resource_type in BLOCK_TYPES:
        return route.abort()
    if any(p in request.url for p in BLOCK_URL_PATTERNS):
        return route.abort()
    return route.continue_()

with sync_playwright() as p:
    browser = p.chromium.launch(
        proxy={"server": "http://proxy.thordata.com:24125",
               "username": "thor-user-pass-pw-sessid-cost01-geo-us",
               "password": "pw"})
    ctx = browser.new_context()
    ctx.route("**/*", lean_page)
    page = ctx.new_page()
    page.goto("https://example.com/products", wait_until="domcontentloaded")
    data = page.eval_on_selector_all(
        ".product-card", "els => els.map(e => e.innerText)")
    browser.close()

Route-blocking images, fonts, media, and trackers through the proxy typically cuts rendered-page bandwidth by 50–70% with zero data loss — the DOM and the data payloads still load. One caveat: some anti-bot systems check whether images were fetched. If you see survival drop after aggressive blocking, unblock images (keep fonts/media/tracker blocking) and re-measure; the balance is target-specific.

Optimization 3: Cache Aggressively, Correctly

Every byte fetched twice through a metered proxy is pure loss. The cache hierarchy for a scraper:

URL-level response cache with TTLs per content class. This is where the data modeling saves money: different page types change at different rates, and your collection frequency should match.

import hashlib, json, time, pathlib

CACHE = pathlib.Path("resp_cache"); CACHE.mkdir(exist_ok=True)
TTL_BY_CLASS = {
    "product_page":  6 * 3600,   # prices: change hourly-ish
    "category_page": 2 * 3600,   # listings: faster churn
    "review_page":  24 * 3600,   # reviews: slow
    "about_page":  7 * 24 * 3600,# static: barely ever
}

def cached_get(session, url, page_class):
    key = hashlib.sha1(url.encode()).hexdigest()
    meta_p, body_p = CACHE / f"{key}.json", CACHE / f"{key}.html"
    if meta_p.exists():
        meta = json.loads(meta_p.read_text())
        if time.time() - meta["ts"] < TTL_BY_CLASS[page_class]:
            return body_p.read_text()          # zero proxy bytes
    r = session.get(url, timeout=30)
    meta_p.write_text(json.dumps({"ts": time.time(), "url": url}))
    body_p.write_bytes(r.content)
    return r.text

Conditional requests where the site supports them: store ETag/Last-Modified, send If-None-Match/If-Modified-Since. A 304 response is tens of bytes instead of tens of kilobytes. APIs with ETag support make this trivial, and it's the most underused free win in the whole playbook.

Deduplication across URLs. Parameterized URLs create phantom duplicates (?utm_source=..., session IDs, sorting variants). Normalize URLs before caching, or you'll re-fetch the same content under a dozen keys. Canonicalize by stripping tracking params and normalizing sort/filter defaults.

Optimization 4: Compress and Trim at the Protocol Level

Three cheap settings: (1) always send Accept-Encoding: gzip, br and verify compression is actually happening (some request libraries don't advertise it by default); Brotli on text-heavy HTML is typically 20–25% smaller than gzip. (2) Use HTTP/2 or HTTP/3 where available — header compression alone saves 1–2 KB per request, which on a 30 KB page is real money at scale. (3) For APIs, request partial fields where supported (many offer fields= or GraphQL selection sets) — fetching two fields instead of forty can be a 90% reduction on that endpoint.

Optimization 5: Collect on Demand, Not on Schedule

The deepest cost lever is not fetching things that didn't change. Full-site crawls on a cron are the expensive habit. The alternatives, in increasing sophistication:

• Conditional collection: only fetch a product page when a cheaper signal (the category page's price snippet, a stock-status endpoint, a sitemap lastmod) indicates it moved.
• Change-driven discovery: watch sitemaps and feeds for new/changed URLs; scrape what changed, not everything.
• Tiered frequency: collect the 5% of pages that drive 95% of decisions (top sellers, key competitors) hourly; everything else daily or weekly. This is the freshness-tiering idea from data engineering, applied to your bandwidth budget — and it usually cuts volume 3–10x for near-zero information loss.

One More Lever: Pool Tiering by Byte Value

Since bandwidth is the billed unit, tiering your proxy pool by bytes-per-task is as important as tiering by target difficulty. The combination that keeps bills sane: run your high-volume, cache-friendly, low-difficulty fetches (category pages, sitemaps, API endpoints on lightly protected sites) through datacenter exits, and reserve expensive residential or mobile bandwidth for the small subset of requests that genuinely need high-trust IPs. If 80% of your bytes flow through a pool that costs a tenth as much, your blended cost per GB drops by most of that difference — with zero change in collection quality, because you're matching spend to difficulty per request, not per project. Review the split monthly; as sites tighten defenses you'll shift more traffic up-tier, and that shift should be a conscious decision, not a surprise on the invoice.

The Meta-Point: Cost Is an Architecture Property

Per-GB pricing turns engineering care into a line item you can watch move. The pipelines with the best margins aren't the ones with the cleverest scraping tricks — they're the ones that (a) fetch the smallest representation of the data (API over HTML over rendered page), (b) never fetch anything twice, and (c) fetch only what changed. A useful exercise: compute your cost per 1,000 records collected, trend it weekly, and treat any jump as an architectural smell. Bytes are the easiest thing in your pipeline to waste and the easiest thing to fix — the savings compound daily, forever.

Disclosure: I use Thordata's metered residential proxies for the bandwidth-conscious collection described in this post. If you want to try them, they're at thordata.com, and the code **thor020* gets you 10% off.*