If your monitor re-crawls 5,000 product pages every hour and ninety percent of them are unchanged, you are paying retail for bytes you already own. Residential proxy billing follows transferred volume, so the naive re-crawl loop pays full price for every page body, every cycle, forever. The fix is not to stop checking freshness — it is to make the check cheap. Make your crawler revalidation-aware: store each URL's ETag and Last-Modified, send If-None-Match / If-Modified-Since on the next pass, and let the origin answer 304 Not Modified. A 304 carries no body; you pay for a header-sized exchange instead of a full page. The bytes you never pull are the cheapest ones.
What the meter actually counts
On a per-GB residential plan, the charge is bytes crossing the exit, request headers and response headers included. Page bodies dominate that total by orders of magnitude. A monitor's whole job is re-observing state, and most observed state does not change between cycles: prices are sticky, rank positions are lazy, review counts creep. A plain local cache with a TTL avoids the request entirely, but it trades correctness for bytes — you might act on hour-old data. Revalidation is the different mechanism: the request still goes out through the proxy, the origin still confirms freshness, but when nothing changed, almost nothing comes back. You keep the correctness and drop the payload.
The two validators, and what each guarantees
An ETag is an opaque quoted string chosen by the origin — a content hash, a version counter, a database revision. You never parse it; you echo it back verbatim in If-None-Match. There are two flavors. A strong validator looks like "a1b2c3"; a weak one is W/"a1b2c3". Only strong validators promise byte-for-byte identical bodies. A weak ETag claims semantic equivalence, so a CDN that varies content by Accept-Encoding, or a page that regenerates a build timestamp, may legitimately hand you a different body with the same W/ tag. For change-detection pipelines that is fine — your parser diffs the payload anyway — but if you hash bodies downstream for dedup or forensic capture, prefer strong validators and know which one you stored. W/ tags are still perfectly valid for conditional GETs; just never strip the W/ prefix yourself when replaying it, because that silently upgrades a weak claim to a strong one and some origins will then answer 200 forever.
Last-Modified is the fallback: an HTTP date with one-second granularity at best, sometimes floored to minutes, sometimes stamped fresh on every render. Send both headers when you have both. Servers that support ETags evaluate If-None-Match first; origins that never emit ETags are still covered by If-Modified-Since.
A revalidation store you can actually ship
The store is small enough to build in an afternoon. SQLite keeps it crash-safe across crawler restarts:
import sqlite3
import time
from dataclasses import dataclass
import requests
PROXIES = {
"http": "http://user:pass@residential-gw.example:33333",
"https": "http://user:pass@residential-gw.example:33333",
}
@dataclass
class FetchResult:
url: str
status: int
body: str | None
revalidated: bool # True = a 304, body served from the store
wire_bytes: int # bytes this call actually cost us
class RevalidationStore:
def __init__(self, path: str = "revalidation.db"):
self.db = sqlite3.connect(path)
self.db.execute(
"""CREATE TABLE IF NOT EXISTS entries (
url TEXT PRIMARY KEY,
etag TEXT,
last_modified TEXT,
fetched_at REAL,
body TEXT,
revalidate INTEGER DEFAULT 1)""")
self.db.commit()
def get(self, url):
return self.db.execute(
"SELECT etag, last_modified, body, revalidate "
"FROM entries WHERE url = ?", (url,)).fetchone()
def put(self, url, etag, last_modified, body):
self.db.execute(
"""INSERT INTO entries VALUES (?, ?, ?, ?, ?, 1)
ON CONFLICT(url) DO UPDATE SET etag = excluded.etag,
last_modified = excluded.last_modified,
fetched_at = excluded.fetched_at, body = excluded.body""",
(url, etag, last_modified, time.time(), body))
self.db.commit()
def disable(self, url):
self.db.execute(
"UPDATE entries SET revalidate = 0 WHERE url = ?", (url,))
self.db.commit()
def conditional_get(url, store, session=None):
session = session or requests.Session()
headers = {"User-Agent": "monitor-bot/1.0"}
cached = store.get(url)
if cached and cached[3]:
if cached[0]:
headers["If-None-Match"] = cached[0] # verbatim, W/ and all
if cached[1]:
headers["If-Modified-Since"] = cached[1]
resp = session.get(url, headers=headers, proxies=PROXIES, timeout=15)
wire = len(resp.content) + sum(len(k) + len(v) + 4
for k, v in resp.headers.items())
if resp.status_code == 304 and cached:
return FetchResult(url, 304, cached[2], True, wire)
if resp.status_code == 200:
if "no-store" not in resp.headers.get("Cache-Control", ""):
store.put(url, resp.headers.get("ETag"),
resp.headers.get("Last-Modified"), resp.text)
return FetchResult(url, 200, resp.text, False, wire)
return FetchResult(url, resp.status_code, None, False, wire)
The request always goes on the wire — that is the point — but it carries stored validators. A 304 returns the cached body and flags revalidated; a 200 refreshes the store. The wire_bytes field feeds your own telemetry so savings are measured, not assumed.
The trap: origins that mint a fresh ETag every response
Pages that embed a request nonce, a session token, or a render timestamp produce a new ETag on every hit. Revalidation against them never yields a 304; you pay full body cost plus the bookkeeping forever. This is detectable, so instrument it:
stats = {} # origin -> [conditional_requests, got_304]
def note(origin, sent_validator, status):
if sent_validator:
s = stats.setdefault(origin, [0, 0])
s[0] += 1
if status == 304:
s[1] += 1
def should_revalidate(origin, min_samples=25, min_rate=0.05):
sent, hits = stats.get(origin, (0, 0))
if sent < min_samples:
return True # not enough evidence yet
return hits / sent >= min_rate # then store.disable(url)
A 5% hit-rate floor over 25 samples is deliberately lenient; the min_samples gate keeps a cold store from being misread as a dead origin. When an origin fails repeatedly, stop sending conditional headers to it and fall back to fetch-and-diff. Revalidation is an optimization, not a religion.
Cache-Control: which directives bless this, which forbid it
no-cache sounds hostile and is actually your best friend: it permits storing the response but requires revalidation before reuse — exactly the conditional GET above. no-store is the real veto: do not persist that body; the next pass pays full price, which is what the origin asked for. private restricts shared caches such as proxies and CDNs; your client-side store is the browser-equivalent cache, so private does not block revalidation. Vary is the quiet landmine: if a response varies on Accept-Encoding or User-Agent and your next pass sends different headers, the origin may answer 200 forever. Key stored entries on the Vary dimensions, or simply freeze your request headers per URL.
The accounting
Model one monitoring cycle: 2,000 URLs, a 470 KB average page, 90% unchanged between cycles, and a $7.50/GB residential rate:
def crawl_economics(n_urls=2000, page_bytes=470_000, unchanged_ratio=0.90,
price_per_gb=7.50, header_cost_bytes=420):
gb = 1024 ** 3
changed = int(n_urls * (1 - unchanged_ratio))
plain = n_urls * (page_bytes + header_cost_bytes)
reval = changed * (page_bytes + header_cost_bytes) \
+ (n_urls - changed) * header_cost_bytes
print(f"without revalidation: {plain / gb:.3f} GB "
f"${plain / gb * price_per_gb:.2f}/cycle")
print(f"with revalidation: {reval / gb:.3f} GB "
f"${reval / gb * price_per_gb:.2f}/cycle")
print(f"savings: {(1 - reval / plain) * 100:.1f}%")
crawl_economics()
# without revalidation: 0.876 GB $6.57/cycle
# with revalidation: 0.088 GB $0.66/cycle
# savings: 89.9%
Run this crawl hourly and the month costs $4,732 unconditionally, or $477 with revalidation — same freshness guarantees either way. The discount tracks the unchanged ratio directly. Static sites, sticky prices, lazy rankings: the flatter the content, the closer revalidation gets to free.
What a 304 still costs you
Be honest in your dashboards: a 304 is still a full round trip. You still pay for the request headers crossing the exit, you still consume a request and an allocated session against your pool, and the origin still answers. What you stop paying for is the body — and on per-GB billing, the body is essentially the bill. Side effect worth noting: a 304 usually short-circuits before template rendering and database reads, so origins tend to like conditional crawlers noticeably more than full-page ones. Politeness and cost finally point the same direction.
Take it to the wire
Take any monitor you own that re-fetches on a schedule and wire in the store tonight. Log the 304 rate per origin for a week, disable revalidation where it never pays, honor no-store, and watch the byte meter decelerate while freshness stays exactly where it was. The cheapest residential gigabytes are the ones you negotiate out of existence.
Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele
Top comments (0)