Cache Before You Route: The Cheapest Proxy GB Is the One You Never Fetch

Proxy cost optimization advice usually starts and ends with shopping: compare per-GB prices, negotiate committed volumes, switch providers. That advice optimizes the unit price of a resource your architecture is wasting. Before you can optimize what you pay per gigabyte, you have to ask how many of those gigabytes needed to exist at all.

Here's the uncomfortable arithmetic. A price-monitoring crawler that checks 50,000 SKUs every 15 minutes fetches roughly 4.8 million pages a day. At ~400 KB per product page, that's close to 2 TB of residential traffic monthly at full price — for a dataset where, on any given cycle, 95%+ of the pages have not changed since the last fetch. You are paying residential rates to re-download bytes you already own.

The core idea of this post: in a well-architected pipeline, the proxy is the last hop, not the first. Put an HTTP-conditional caching layer in front of proxy routing and let ETag/Last-Modified do the work. A 304 response is a few hundred bytes through a tunnel instead of 400 KB, and your bandwidth bill tracks actual change, not fetch frequency.

The three layers of "don't fetch it again"

Layered cheapest-first, because order is everything:

1. Deduplication at the scheduler: if the URL was fetched successfully 40 seconds ago in another worker, don't fetch it again. Boring, but the biggest single win in most codebases I audit — parallel workers independently crawling overlapping URL sets is endemic.
2. Conditional requests (this post's focus): send If-None-Match/If-Modified-Since with the validator you stored from the last fetch. A 304 costs a round trip but almost no bytes.
3. Content-hash skipping: even on a 200, hash the extracted payload — if the fields you actually store (price, stock, title) are unchanged, downstream processing and storage skip, and you learn the page's real change rate for tuning.

Layers 1 and 3 are bookkeeping. Layer 2 is where the proxy economics live, so let's build it properly.

A conditional cache in front of the proxy

The cache maps URL -> (etag, last_modified, fetched_at, body_hash). On every fetch it attaches validators; on a 304 it returns the stored body. Persistence is a JSON file for demonstration — in production, SQLite or Redis.

import requests
import hashlib
import json
import time
import os

class ConditionalCache:
    """HTTP conditional caching layer that sits BEFORE proxy routing."""

    def __init__(self, username, password, path="cache.json",
                 fresh_window=90, stale_max=86400):
        """
        fresh_window: seconds during which we don't even revalidate.
        stale_max:    seconds after which a cached entry is unusable.
        """
        self.path = path
        self.fresh_window = fresh_window
        self.stale_max = stale_max
        self.store = self._load()
        self.proxies = {
            "http": f"http://{username}:{password}@resi.thordata.com:9001",
            "https": f"http://{username}:{password}@resi.thordata.com:9001",
        }

    def _load(self):
        if os.path.exists(self.path):
            with open(self.path) as f:
                return json.load(f)
        return {}

    def _save(self):
        with open(self.path, "w") as f:
            json.dump(self.store, f)

    def fetch(self, url: str, force=False) -> dict:
        entry = self.store.get(url)
        now = time.time()

        # 1) Fresh enough: don't touch the network at all.
        if not force and entry and now - entry["fetched_at"] < self.fresh_window:
            return {"status": "fresh-hit", "body": entry["body"],
                    "bytes_via_proxy": 0}

        # 2) Stale beyond usefulness: treat as miss (drop validators).
        if entry and now - entry["fetched_at"] > self.stale_max:
            entry = None

        headers = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)"}
        if entry:
            if entry.get("etag"):
                headers["If-None-Match"] = entry["etag"]
            if entry.get("last_modified"):
                headers["If-Modified-Since"] = entry["last_modified"]

        r = requests.get(url, proxies=self.proxies, headers=headers, timeout=25)

        if r.status_code == 304 and entry:
            # The win: ~300 bytes through the tunnel instead of ~400 KB.
            entry["fetched_at"] = now
            self._save()
            return {"status": "not-modified", "body": entry["body"],
                    "bytes_via_proxy": len(r.content)}

        r.raise_for_status()
        entry = {
            "body": r.text,
            "etag": r.headers.get("ETag"),
            "last_modified": r.headers.get("Last-Modified"),
            "body_hash": hashlib.sha256(r.content).hexdigest(),
            "fetched_at": now,
        }
        self.store[url] = entry
        self._save()
        return {"status": "fetched", "body": entry["body"],
                "bytes_via_proxy": len(r.content)}

Three details separate this from a naive implementation. First, fresh_window — revalidating every cycle is itself wasteful when the target's change cadence is slower than your crawl cadence. Second, the stale_max guard — validators from yesterday's CDN config can cause weird 200s-with-junk; expire them. Third, body_hash in the entry: it powers layer 3, because a 200 can still be a no-change event.

Measuring the actual saving: GB accounting per fetch

You can't optimize what you don't meter, so make the cache report its own economics:

class GBLedger:
    def __init__(self):
        self.bytes_by_status = {}

    def record(self, result: dict):
        status = result["status"]
        self.bytes_by_status[status] = (
            self.bytes_by_status.get(status, 0) + result["bytes_via_proxy"])

    def report(self, price_per_gb=3.5):
        total = sum(self.bytes_by_status.values())
        print(f"{'status':14} {'MB':>10}")
        for k, v in sorted(self.bytes_by_status.items()):
            print(f"{k:14} {v/1e6:>10.1f}")
        cost = total / 1e9 * price_per_gb
        print(f"total: {total/1e6:.1f} MB  ~ ${cost:.2f} at ${price_per_gb}/GB")
        full_refetch = None
        return total

ledger = GBLedger()
cache = ConditionalCache("user", "pass")
url = "https://shop.example.com/product/B08N5WRWNW"

for cycle in range(10):          # 10 crawl cycles, e.g. every 15 min
    result = cache.fetch(url)
    ledger.record(result)
ledger.report()

Run this against a real target and the pattern is dramatic: cycle 1 shows fetched with ~400 KB, cycles 2-10 show not-modified at a few hundred bytes each. On a catalog where 5% of pages change per cycle, bandwidth through the proxy drops by roughly an order of magnitude. That's the entire optimization — no vendor negotiation required.

When targets don't cooperate (and what to do)

The honest caveat: not every target emits ETag or honors conditional requests. E-commerce product pages behind edge caches usually do; script-rendered SPAs frequently don't. Audit your targets first:

def validator_audit(urls, proxies):
    for url in urls:
        r = requests.get(url, proxies=proxies, timeout=25)
        etag = bool(r.headers.get("ETag"))
        lm = bool(r.headers.get("Last-Modified"))
        cc = r.headers.get("Cache-Control", "")
        print(f"{url}\n  ETag={etag} Last-Modified={lm} Cache-Control={cc!r}")

validator_audit(["https://shop.example.com/p/1"], cache.proxies)

When validators are absent, fall back to layer 3 — you must fetch, but hash the extracted fields and skip everything downstream when unchanged, then let the measured change rate set your crawl cadence. A SKU that has changed twice in 90 days does not need 15-minute polling; adaptive per-URL intervals (crawl frequently after a change, back off exponentially during stability) compound with conditional requests and routinely cut another 30-50% of fetch volume.

One warning about cleverness: never serve stale data past the point where your consumers notice. The fresh_window and stale_max bounds exist to keep the cache honest. A monitoring product that reports yesterday's price as current is saving money by being wrong, which is the most expensive optimization there is.

One scaling note: the cache's value compounds with worker count. A single worker re-fetching out of ignorance wastes one request; fifty workers doing it concurrently waste fifty, and — worse — they arrive at the target as a synchronized burst from your exit IPs, which is exactly the traffic shape bot defenses are built to notice. Moving the cache from per-process memory to a shared store (Redis works well: the validator entry is small, and a GET url before every fetch is negligible next to a proxied round trip) turns those fifty redundant fetches into one fetch plus forty-nine lock-waits. The proxy sees a steady, modest stream instead of a thundering herd, and your spend drops in the same motion. Cost optimization and politeness optimization are, at the architecture level, the same optimization.

Where this fits in the stack

The mental model that ties it together: your pipeline has a request waterfall — scheduler → dedup → conditional cache → proxy router → target. Cost lives at the proxy hop; intelligence should live upstream of it. Most teams invert this: their schedulers are naive and their cost discussions are all about the proxy hop, so they optimize the most expensive tier to fetch bytes they already have. Flip the order. Dedup decides, the cache validates, the proxy fetches only what's genuinely new, and the per-GB bill becomes a function of your dataset's real change rate — which is the only denominator it should ever have had.

---

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele