Feeding AI Agents Fresh Web Data: A Practical Pipeline

AI agents are only as current as their data. An agent that answers "what does this product cost right now" from a stale snapshot isn't an agent — it's a cache with a personality.

The gap between "LLM trained last quarter" and "question asked today" is a data engineering problem, and it's becoming the data engineering problem as teams bolt retrieval onto models. This post lays out a practical pipeline for keeping AI agents fed with fresh, clean web data.

The Shape of the Problem

Three properties make web data uniquely annoying for AI systems:

Volatility — prices change daily, availability hourly, rankings by the minute. A snapshot has a half-life measured in days.

Variety — the sources are hostile HTML, not tidy APIs. Every integration is bespoke.

Vantage dependence — the same URL returns different content by visitor geography. A "global" dataset collected from one IP location is quietly a single-market dataset.

A pipeline that handles all three has four stages: collect, normalize, store, serve. The interesting parts are the first and last.

Stage 1: Collection That Doesn't Rot

The collector must run continuously without getting blocked, because a blocked collector produces silently stale data — worse than an error, because nothing announces it.

The blocking-resistance stack, in order of leverage:

1. Residential IPs — datacenter ranges start every request with a guilty score
2. Geo-targeting — each market's data fetched from in-country IPs, so you get that market's actual content
3. Sticky sessions for authenticated sources — consistent identity per login
4. Politeness — jittered delays, per-domain rate ceilings, retries with backoff

def collect_market(market, urls):
    for url in urls:
        proxy = {
            "http": f"http://{USER}-country-{market}:{PASS}@gate.thordata.com:9000",
        }
        r = requests.get(url, headers=HEADERS, proxies=proxy, timeout=30)
        store_raw(r.text, market=market, url=url, ts=utcnow())
        time.sleep(random.uniform(1.5, 4.0))

Note store_raw — always keep the raw HTML. Parsing rules change; you don't want to re-collect history to fix a parser bug.

Stage 2: Normalization (Where LLMs Genuinely Help)

Here's where modern pipelines diverge from classic ETL: extraction itself can be model-driven. Instead of maintaining brittle CSS selectors per source, hand the cleaned HTML to a small fast model with a schema:

from pydantic import BaseModel

class Product(BaseModel):
    name: str
    price: float
    currency: str
    availability: str

resp = client.models.generate_content(
    model="gemini-2.5-flash",  # or any fast cheap extractor
    contents=f"Extract: {strip_scripts(raw_html)}",
    config={"response_mime_type": "application/json",
            "response_schema": Product},
)
record = Product.model_validate_json(resp.text)

The economics work because extraction is a per-page cost measured in fractions of a cent, and the schema guarantees valid JSON. Two hard-won tips: strip scripts/styles before sending (60–70% token reduction), and require an "evidence" field you can substring-verify against the raw HTML to catch hallucinated values.

Stage 3: Storage With Time Built In

The schema that has survived every pipeline I've built:

records(source, market, url, extracted JSONB, collected_at)

Everything is append-only. "Current state" is a view: latest record per (source, url, market). "History" falls out for free, and history is what powers trend questions — "is this price rising?" — that agents get asked constantly.

Stage 4: Serving It to Agents

The interface between pipeline and agent is usually one of:

• A retrieval layer (vector store for semantic lookups) — right for "find products matching X"
• A simple query API (GET /price?url=...) — right for tool-calling agents, and honestly underrated
• A freshness-limited cache — the agent tool checks collected_at and triggers re-collection if stale

The last one matters: give your agent a refresh action, not just a read. An agent that can notice its price data is 6 hours old and pull a fresh fetch (through the same geo-targeted collector) closes its own loop.

Failure Modes That Sneak Up

Silent staleness. Your collector gets soft-blocked and starts receiving CAPTCHA pages; your parser extracts garbage or nothing; the "current" view quietly freezes at last week. Monitor collection success rate as a first-class metric, and alert when it drops.

Vantage drift. Data collected from a proxy pool whose geo mix silently changes (a provider rebalancing exit nodes) shifts your dataset's composition over time. Log collection_country per record and check the distribution weekly.

Hallucinated extraction. The model-side extractor occasionally invents plausible values. The evidence-verification pattern above is the cheapest defense.

Cost inversion. If collection costs exceed model inference costs, your unit economics broke somewhere — usually too-frequent polling of pages that rarely change. Poll proportionally to volatility.

The Bottom Line

Fresh web data for AI agents is a continuous system, not a dataset. The teams that win at it treat the collector like production infrastructure (geo-matched residential IPs, monitored success rates) and the extraction like a model problem (schema-guaranteed, evidence-verified). Everything between is plumbing.

---

Disclosure: my collection layer runs on Thordata's geo-targeted residential proxies (100M+ IPs, 190+ countries; rotating from $0.65/GB, code thor020 for 10% off). The architecture is provider-independent.