Data Freshness Strategies for ML Pipelines: Tiering Collection by Change Frequency

Ask an ML engineer how fresh their web-scraped training data should be and you'll get the same answer every time: "as fresh as possible." It sounds rigorous. It's actually a category error — because "as fresh as possible" treats freshness as free.

It isn't. Freshness is bought with fetch volume, proxy bandwidth, parsing cycles, and — the currency nobody budgets — detection risk. Every unnecessary fetch is a chance to get blocked, which degrades the freshness of everything else that source provides. I've watched pipelines get less fresh overall because they demanded maximal freshness on everything: the target site noticed the volume, throttled the scraper, and the truly volatile data started arriving late.

The claim of this post: freshness should be allocated in proportion to how fast the data actually changes — not uniformly, and not maximally. Uniform freshness is the silent budget-waster of data engineering. Here's how to do it properly.

Step 1: Accept That Not All Data Decays at the Same Speed

Any web-derived dataset is a mixture of change frequencies. Take a product dataset from an e-commerce scrape: stock counts change by the minute; prices change a few times a day; product descriptions change monthly; category trees change quarterly; the store's layout changes whenever someone in marketing gets ideas.

If you re-scrape everything hourly, you're paying hour-freshness prices for data with a half-life of weeks — and creating 100x more traffic than the information content justifies. If you re-scrape everything daily, your stock counts are useless and your price signals arrive after the market has moved.

The fix is stratification: classify every field and every source by its observed change frequency, and assign each tier its own collection cadence.

Step 2: Build the Tier Model

Start with priors from domain knowledge, then let data correct them:

from dataclasses import dataclass
from datetime import timedelta

@dataclass
class FreshnessTier:
    name: str
    max_age: timedelta        # staleness budget
    poll_interval: timedelta # how often we re-check

TIERS = {
    # priors — refine from observed change data (step 3)
    "realtime": FreshnessTier("realtime", timedelta(minutes=15),
                              timedelta(minutes=15)),
    "volatile": FreshnessTier("volatile", timedelta(hours=1),
                              timedelta(hours=1)),
    "daily":    FreshnessTier("daily", timedelta(days=1),
                              timedelta(hours=12)),
    "weekly":   FreshnessTier("weekly", timedelta(days=7),
                              timedelta(days=2)),
    "static":   FreshnessTier("static", timedelta(days=30),
                              timedelta(days=7)),
}

# field-level assignment for an e-commerce source
FIELD_TIERS = {
    "stock_count": "realtime",
    "price":       "volatile",
    "rating":      "daily",
    "description": "weekly",
    "category":    "static",
}

Two distinct concepts here, and conflating them causes subtle bugs. Poll interval is how often you go look. Max age (the staleness budget) is what you promise downstream consumers — the SLA. Poll interval is always ≤ max age, because you need slack to survive failures: if you poll at exactly your staleness budget, one failed fetch blows the SLA. A good rule: poll at half your max age. A tier promising one-hour-fresh data gets a ~30-minute cycle, so two consecutive failures still land inside budget.

Step 3: Let Observed Change Rates Correct Your Priors

Priors are where you start, not where you stay. Once you have history, measure actual per-item change frequency and re-tier accordingly:

from collections import defaultdict
from datetime import datetime, timedelta

class ChangeProfiler:
    """Tracks observed change rate per item; suggests tier demotions."""

    def __init__(self):
        self.checks = defaultdict(int)    # item -> times checked
        self.changes = defaultdict(int)   # item -> times value differed

    def record(self, item, new_value, old_value):
        self.checks[item] += 1
        if new_value != old_value:
            self.changes[item] += 1

    def effective_tier(self, item, min_checks=20):
        n = self.checks[item]
        if n < min_checks:
            return None                    # not enough evidence yet
        rate = self.changes[item] / n      # fraction of checks with a change
        if rate < 0.01:  return "static"
        if rate < 0.05:  return "weekly"
        if rate < 0.30:  return "daily"
        return "volatile"

This is where the real savings appear. In one pricing dataset I ran, profiling revealed that 63% of "daily-tier" items hadn't changed in over a month. Demoting them to weekly cut total fetch volume by 40% with zero measurable impact on downstream accuracy — because we weren't skipping changed data, we were skipping unchanged data that we'd been re-fetching on faith.

The profiler also catches the opposite: items changing faster than their tier assumes. A price that changes every check while polled daily is data you're systematically sampling too slowly — the tier promotion matters more than the demotions.

Step 4: Cheap Change Detection Before Expensive Fetching

Tiered polling is half the strategy. The other half: don't fetch content you can cheaply prove hasn't changed. Use the web's own staleness signals as a pre-filter:

import hashlib, requests

class ConditionalFetcher:
    """Skips fetches when the source says nothing changed."""

    def __init__(self):
        self.session = requests.Session()
        self.etags = {}        # url -> etag
        self.lastmod = {}      # url -> last-modified header

    def fetch_if_changed(self, url):
        headers = {}
        if etag := self.etags.get(url):
            headers["If-None-Match"] = etag
        if lm := self.lastmod.get(url):
            headers["If-Modified-Since"] = lm

        r = self.session.get(url, headers=headers, timeout=30)
        if r.status_code == 304:
            return None                       # not modified — free skip
        r.raise_for_status()
        if etag := r.headers.get("ETag"):
            self.etags[url] = etag
        if lm := r.headers.get("Last-Modified"):
            self.lastmod[url] = lm
        return r.text

ETag/If-None-Match and Last-Modified/If-Modified-Since turn re-validation into a 304 round-trip — a few hundred bytes instead of a full page fetch. Many APIs expose equivalents (updated_at fields, webhook subscriptions, since parameters). Sitemaps with lastmod timestamps let you skip whole swathes of untouched URLs before any page fetch. For HTML without header support, a HEAD-then-GET pattern or content-hash comparison over a light index page achieves a similar effect. None of these are free, but they're all cheaper than the full fetch + parse + detection risk.

Step 5: Freshness Contracts for Downstream Consumers

Tiering only works if downstream knows what freshness each field carries. A model trained on mixed-age data needs the ages to be features or at least filters — training a price model on stock counts that are up to 24 hours stale while prices are 1-hour fresh bakes a systematic skew into the dataset.

So make freshness a first-class part of the data contract:

@dataclass
class Observation:
    value: object
    observed_at: datetime
    tier: str
    is_stale: bool            # observed_at older than tier.max_age

def validate_batch(batch: list[Observation]) -> list[str]:
    errors = []
    for obs in batch:
        if obs.is_stale and obs.tier in ("realtime", "volatile"):
            errors.append(f"{obs.tier} observation is stale — block, don't serve")
    return errors

Two rules make the contracts real. First: stale realtime-tier data should block, not serve — a 3-hour-old stock count is worse than no data, because it's confidently wrong. Second: record the age at training time. Staleness correlates with real-world signal (old prices during volatile periods are differently informative), and models that see the age learn to discount appropriately.

The Scheduler That Ties It Together

The final piece is a scheduler that fans each tier out on its own cadence, spreading load within the window:

import random, time

def schedule_tier(tier: FreshnessTier, items: list, fetch_fn):
    interval = tier.poll_interval.total_seconds()
    spacing = interval / max(len(items), 1)
    for item in items:
        fetch_fn(item)                       # respects per-domain politeness
        time.sleep(spacing * random.uniform(0.8, 1.2))  # jitter: no spikes

Jittered spacing within each tier's window matters more than it looks: starting every hourly fetch in one burst creates a traffic spike that's both detectable and wasteful of your own concurrency. Spread each tier's work across its window and your request stream becomes near-uniform at a much lower average rate — which is exactly the traffic profile that never gets throttled.

Wrapping Up

"Maximal freshness" isn't a strategy, it's an abdication of one. The mature approach: stratify data by observed change frequency, poll each tier at half its staleness budget, use cheap change-detection to skip unchanged sources, and expose per-field freshness contracts so downstream models know exactly how old everything is. You get more useful freshness where it matters — because you stopped burning your fetch budget and detection capital on data that never changes.

Disclosure: I use Thordata's residential proxies for the geo-pinned fetching in these tiered pipelines — the per-tier scheduler leans on steady, session-sticky exits. They're at thordata.com, and the code **thor020* gets you 10% off.*

Comments

[Luis Cruz]

I appreciate your insight into the cost of data freshness in ML pipelines, particularly the emphasis on tailoring refresh rates to actual change frequencies. The stratification approach you outlined could significantly optimize resource usage while maintaining the quality of critical data. It might also be useful to consider implementing a feedback loop that dynamically adjusts the tiers based on real-time data volatility, which could enhance responsiveness to sudden market shifts. If you're exploring ways to refine this tier model or need additional engineering support, I’d be glad to discuss a paid collaboration.