Designing AI agents that can use live web data

An AI agent that reads stale data is worse than one that admits it cannot answer. Pricing, listings, reviews, inventory, job posts, and product availability change constantly, but many agent workflows still depend on CSV exports, copied spreadsheets, or APIs that only cover part of the job.

The hard part is not asking the model to reason over web data. The hard part is getting current, structured data into the agent in a way that fails visibly when the source changes.

Treat web access as a data contract

Do not give an agent a vague tool called browse_web and hope it figures out the rest. Define the shape of the data you expect back.

For example, if you are collecting competitor hotel prices, the agent probably needs this:

{
  "source_url": "https://example.com/search?city=miami",
  "fetched_at": "2026-08-18T10:15:00Z",
  "currency": "USD",
  "results": [
    {
      "name": "Hotel Example",
      "nightly_rate": 184,
      "available": true,
      "rating": 4.6,
      "review_count": 812
    }
  ]
}

That schema matters because it gives you something to validate before the model touches the data. If results is empty, that might mean no availability. It might also mean the scraper hit a bot page, the DOM changed, or the site returned a 403. Those cases should not collapse into the same answer.

A catalogued extractor such as Wire can help here because the agent calls a named action with known parameters instead of inventing selectors at runtime.

Pick the extraction method based on how the site works

There are a few common ways to get live web data. They fail differently.

Plain HTTP works when the content is in the HTML response:

curl -L https://example.com/pricing

This is cheap and easy to cache, but it breaks when the page renders data client-side.

Browser rendering works when the site needs JavaScript:

import { chromium } from 'playwright';

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto('https://example.com/search?q=miami', {
  waitUntil: 'networkidle'
});

const cards = await page.locator('[data-testid="listing-card"]').allTextContents();
await browser.close();

This handles more sites, but it costs more per request and tends to fail under volume. You also inherit timing problems: a selector exists locally, then fails in production because a cookie banner, A/B test, or slow API call changed what rendered.

Network-level extraction is often better for listings and search pages. Many sites load data by calling JSON endpoints from their own frontend. If you can identify that request, you can call it directly or replay the same parameters. You avoid spinning up a browser and parsing HTML.

For high-volume listing pages, Wire is one example of this network-call approach: capture the background API calls the frontend already uses and return structured records instead of rendered markup.

Authenticated pages are a separate case. Do not pass user passwords through the agent. Store a browser session or token outside the model, then expose only the specific action the agent is allowed to perform, such as get_open_invoices or fetch_saved_search_results.

Make scraping asynchronous when it can take time

Live extraction often takes longer than a normal API request. A browser may need to load several pages. A crawl may fan out across a site. A target may rate limit you.

Use a job model instead of forcing the agent to wait on one long request.

import time
import requests

API_KEY = 'replace-me'
BASE_URL = 'https://scraper.internal.example'

headers = {
    'Authorization': f'Bearer {API_KEY}',
    'Content-Type': 'application/json'
}

start = requests.post(
    f'{BASE_URL}/jobs',
    headers=headers,
    json={
        'url': 'https://example.com/search?city=miami',
        'render_js': True,
        'output': 'json'
    },
    timeout=10
)
start.raise_for_status()

job_id = start.json()['id']
deadline = time.time() + 90

while time.time() < deadline:
    res = requests.get(f'{BASE_URL}/jobs/{job_id}', headers=headers, timeout=10)

    if res.status_code == 429:
        time.sleep(5)
        continue

    res.raise_for_status()
    payload = res.json()

    if payload['status'] == 'succeeded':
        data = payload['result']
        if not data.get('results'):
            raise RuntimeError('Scrape succeeded but returned zero results')
        print(data)
        break

    if payload['status'] == 'failed':
        raise RuntimeError(payload.get('error', 'Scrape failed without an error message'))

    time.sleep(2)
else:
    raise TimeoutError(f'Scrape job {job_id} did not finish within 90 seconds')

The important part is not the exact endpoint. It is the contract:

• The first request returns a job ID.
• Polling reports queued, running, succeeded, or failed.
• Failures include a reason the agent can report.
• Timeouts are explicit.
• Empty output is treated as suspicious unless the query expected it.

If you skip this, your agent will eventually produce confident nonsense because the scraper returned a login page, a CAPTCHA, or an empty array.

Return evidence with the data

Agents need more than records. They need enough metadata to know whether the records are usable.

Include fields like:

{
  "source_url": "https://example.com/search?city=miami",
  "fetched_at": "2026-08-18T10:15:00Z",
  "http_status": 200,
  "extractor_version": "hotel-search-v3",
  "result_count": 42,
  "warnings": []
}

Warnings are useful. For example:

{
  "warnings": [
    "Only 8 results returned, expected at least 25 for this market",
    "Prices were missing for 3 listings",
    "Source page showed results in EUR, converted to USD"
  ]
}

That gives the agent room to answer carefully: I found 8 listings, but this looks incomplete, instead of treating partial data as complete.

Keep the agent on rails

The model should decide what data it needs and how to interpret it. It should not decide how to scrape a complex website from scratch on every run.

A better pattern is to expose narrow tools:

type FetchCompetitorRatesInput = {
  city: string;
  checkIn: string;
  checkOut: string;
  guests: number;
};

Behind that tool, you can change selectors, swap browser rendering for network extraction, add retries, or fix rate-limit handling without changing the agent prompt.

Start by picking one workflow where stale web data causes real manual work. Define the output schema, implement the extractor as an async job, and make the failure states visible before you connect it to an agent.