Match the Rotation to the Refresh Rate: A Proxy Policy for Freshness-Tiered Collection

Data-freshness advice almost always talks about the caller: "re-crawl the news page every five minutes, the pricing page hourly, the about-page weekly." That half is easy and everyone gets it. The half nobody writes about is on the wire, not in the scheduler: how fresh your data can be is partly decided by which exit you fetched through and how you treat sessions on that exit. A page that changed sixty seconds ago can still be served to you as a stale, cached, personalized, or A-pinned-to-an-old-IP response, and no scheduler cadence fixes that. The freshness of your corpus is a joint function of request timing and egress behavior.

The thesis: stop tuning only the crawl interval. Classify each target by its true change rate, then bind every class to a proxy policy — tier, rotation cadence, and session-stickiness — that actually delivers fresh bytes. Match the rotation to the refresh rate, and you get freshness you can guarantee rather than hope the CDN felt like it.

Two clocks, and only one is yours

There are two clocks in any freshness problem. The first is the one you control: how often you issue a request. The second is the one that controls what comes back: how long a cached or sticky-pinned representation of the page lives between the origin and your socket. Most "stale data" bugs I have debugged were clock-one people blaming clock-two, or vice versa.

Concretely, a fast-changing page requested every minute can still return a five-minute-old cached response if every one of those requests exits from the same IP and the edge cache keys on that IP. Conversely, a slow-changing page re-fetched through fresh exits every time pays full proxy GB for bytes that never changed. The naive "poll faster" instinct makes freshness and cost worse at the same time when the exit is pinned, and makes cost worse for no freshness gain when the content rarely moves.

The lever that connects the two clocks is session behavior at the exit: sticky vs rotating, and what you keep sticky. That lever is a proxy-pool property, not a crawler property, which is why freshness belongs in the same conversation as egress design.

Tier targets by genuine change rate, not by vibes

A tiering scheme that works in practice buckets each source into a small number of classes, each with a policy attached rather than a single number. Change rate here means how often the content a given exit sees actually differs, which is different from how often the origin edits it: if the edge pins you to yesterday's HTML, your effective change rate is the edge TTL, not the deploy cadence.

A reasonable three-tier policy, expressed as data not prose:

from dataclasses import dataclass


@dataclass(frozen=True)
class FreshnessTier:
    name: str
    poll_seconds: int          # how often YOU re-fetch (your clock)
    max_session_stickiness_s: int   # longest a sticky exit may be reused
    bypass_cache_rotation: bool     # rotate exits to dodge per-exit edge caches
    tier_hint: str             # cost/quality posture


TIERS: dict[str, FreshnessTier] = {
    "hot": FreshnessTier(
        name="hot", poll_seconds=120, max_session_stickiness_s=0,
        bypass_cache_rotation=True, tier_hint="rotating_residential",
    ),
    "warm": FreshnessTier(
        name="warm", poll_seconds=3600, max_session_stickiness_s=600,
        bypass_cache_rotation=False, tier_hint="isp_or_residential",
    ),
    "cold": FreshnessTier(
        name="cold", poll_seconds=86_400, max_session_stickiness_s=86_400,
        bypass_cache_rotation=False, tier_hint="datacenter",
    ),
}

Read the policy as a whole, because the fields trade against each other. Hot pages get a short poll, zero stickiness, forced rotation, and the priciest tier — because for genuinely volatile content, freshness is worth the GB and per-exit caching is the enemy. Cold pages get a daily poll, long-lived sticky sessions, no rotation, and the cheapest tier — because re-hitting a rarely-changing page through fresh residential exits is paying a premium to re-download yesterday. Warm sits in the middle, and most real targets are warm.

Cache-busting rotation is a proxy mechanism, not a query-string hack

The classic way people force fresh content is appending ?_=<timestamp> to the URL. Sometimes that works; often the cache ignores unknown params, or worse, it poisons your dedup key and makes one page look like infinitely many. The cleaner mechanism lives one layer down: change the exit. If the edge keys its cache on client IP, a fresh exit is a cache miss, delivered honestly — you get the same bytes a first-time visitor in that region would get, which is usually the freshest thing on offer.

That is precisely the "hot" tier's bypass_cache_rotation=True. Here is a policy-aware picker that decides whether to reuse the last exit or force a new one, based on how stale the sticky session is allowed to be:

import random
import time
from dataclasses import dataclass, field


@dataclass
class Session:
    exit_ip: str
    created_at: float


@dataclass
class TieredProxyPool:
    tier: FreshnessTier
    exits: list[str]
    _sticky: Session | None = field(default=None, repr=False)

    def get_exit(self) -> str:
        now = time.monotonic()
        reuse_window = self.tier.max_session_stickiness_s

        # Forced rotation: never reuse; each fetch looks like a brand-new visitor.
        if self.tier.bypass_cache_rotation or reuse_window <= 0:
            self._sticky = None
            return random.choice(self.exits)

        # Bounded sticky session: hold the exit to exploit edge caching, but drop
        # it once it can no longer be trusted to have refreshed.
        if self._sticky and (now - self._sticky.created_at) <= reuse_window:
            return self._sticky.exit_ip

        self._sticky = Session(exit_ip=random.choice(self.exits), created_at=now)
        return self._sticky.exit_ip

The subtlety is reuse_window for the warm/cold tiers. There, you want stickiness: a stable exit that the edge associates with a warm cache line returns cheap, consistent bytes for content that is not moving. Stickiness is a freshness liability only for hot content, and a cost saver for cold content. One knob, opposite signs, decided entirely by the tier.

Measure delivered freshness, not just poll interval

You cannot manage the second clock by staring at the first. The metric that matters is how old the bytes actually were when they reached you, and whether successive fetches from the same exit return anything at all. A page you poll every two minutes that returns byte-identical content for six hours is being served stale, and you are paying per-fetch costs for zero new information.

import hashlib


def content_hash(body: str) -> str:
    return hashlib.sha256(body.encode("utf-8", "ignore")).hexdigest()


class FreshnessProbe:
    """Track how often the SAME exit hands you genuinely different content."""

    def __init__(self):
        self.last = {}          # key -> (content_hash, monotonic_time)

    def observe(self, key: str, body: str, now: float) -> dict:
        h = content_hash(body)
        prev = self.last.get(key)
        changed = prev is None or prev[0] != h
        if changed:
            self.last[key] = (h, now)
        return {
            "changed": changed,
            "since_change_s": None if prev is None else now - prev[1],
        }

Feed every fetch through the probe. If a "hot" target shows changed=False across many polls while sitting on one sticky exit, your clock-two is the bottleneck and the fix is to rotate, not to poll harder. If a "cold" target shows changed=True on every fresh exit but the content delta is nil, you are over-rotating and burning residential GB to re-fetch static bytes — the fix is to add stickiness and drop to a cheaper tier. The probe tells you, per source, which lever to pull.

Cost follows the same tier, which is the whole point

There is a reason to be strict about tiering beyond correctness: it is your cheapest cost lever. A typical corpus is power-law in change rate — a handful of hot endpoints, a long tail of pages that basically never move. If you treat everything like the hot tier, you pay residential-plus-rotation prices on content you could have served from a datacenter sticky session once a day. Grade the tail correctly and the bill collapses, because most of your URLs belong in "cold," where freshness is cheap precisely because there is nothing fresh to fetch.

The mistake teams make is one global posture. "We rotate everything, always, through residential, it's safest." Safest for the ten hot pages; ruinous for the ten thousand cold ones. Freshness is a per-tier policy question, and the same tier that guarantees fresh bytes for volatile targets is the tier that lets you stop paying for bytes that do not change. Get the classification right, bind exit behavior to it, and both clocks tick in your favor.

The honest version

Stop treating freshness as a single dial on the crawler. Classify each target by the change rate it actually delivers, then bind that class to a proxy policy: tier, rotation, and stickiness chosen together. Force rotation on hot content to defeat per-exit edge caching and get honest first-visit bytes. Lean on sticky sessions and cheaper tiers for cold content, where a stable exit is a feature, not a bug. And measure delivered freshness — the changed flag, the hash, the seconds-since-last-actual-change — because that is the second clock, and it is the one that was lying to you. Poll interval is easy. Fresh bytes at a price you can defend is the hard part, and it lives in your exit pool.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele