What coding agents need from a real web data layer

You ask a coding agent to compare three pricing pages, check a changelog, or pull current compliance dates. It comes back with something plausible, but one source was stale, another page was JavaScript-rendered, and the third was summarized so aggressively that the important caveat disappeared.

That is usually not a model problem. It is a data access problem.

Built-in web tools are lookup tools

Most agent harnesses have some form of web access now. Claude Code has WebSearch and WebFetch. Codex can run with cached, live, or disabled web search. Cursor and other IDE agents expose similar primitives.

These tools help, but they are not the same thing as a web data layer.

Search usually returns titles, URLs, and maybe short snippets. That is enough to decide what to fetch next, but not enough to reason over the source. Fetch tools often return a model-generated summary instead of the original page content. That keeps token usage down, but it also means the agent reasons over another model's interpretation.

The common failure modes are familiar:

Search result: "Pricing | Example SaaS"
Agent conclusion: "The Pro plan costs $49/month"
Actual page: price changed to $59/month behind client-side rendering
Fetched HTML: <div id="root"></div>

Or this:

HTTP 403
<title>Just a moment...</title>
Cloudflare challenge page returned instead of article content

If your agent cannot tell the difference between a real page and a challenge page, it will often continue anyway.

Treat web access as a small data pipeline

A useful web data layer has three separate jobs:

1. Search and return enough text to rank sources.
2. Fetch the actual page, including JavaScript-rendered content when needed.
3. Convert messy page content into typed data the agent can validate.

I like to make that boundary explicit:

type SearchResult = {
  url: string;
  title: string;
  snippet: string;
  date?: string;
};

type FetchResult = {
  url: string;
  status: "completed" | "failed" | "timeout";
  markdown?: string;
  html?: string;
  error?: string;
};

interface WebDataLayer {
  search(query: string, limit: number): Promise<SearchResult[]>;
  fetchPage(url: string, opts?: { renderJs?: boolean }): Promise<FetchResult>;
  extract<T>(input: string, schemaName: string): Promise<T>;
}

This shape matters because it gives the agent concrete states to handle. A failed fetch is not the same as an empty page. A rendered page is not the same as raw HTML. A snippet is not source content.

Anakin's remote MCP connector follows this split with search, scrape, crawl, and structured site actions, but the pattern is useful even if you build the tools yourself.

Async scraping is usually the right default

Fetching a static HTML page can be synchronous. Rendering a SPA, waiting for network calls, rotating proxies, or retrying after a bot challenge should not block an agent request forever.

A better scraper API returns a job ID, then lets the caller poll:

curl -X POST https://example.internal/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com/pricing","renderJs":true}'

# 202 Accepted
# {"jobId":"job_abc123","status":"pending"}

Then your agent-side wrapper handles completion, failure, and timeout explicitly:

async function waitForScrape(jobId: string, timeoutMs = 60_000) {
  const started = Date.now();

  while (Date.now() - started < timeoutMs) {
    const res = await fetch(`https://example.internal/scrape/${jobId}`);
    const job = await res.json();

    if (job.status === "completed") return job;

    if (job.status === "failed") {
      throw new Error(`Scrape failed: ${job.error ?? "unknown error"}`);
    }

    await new Promise(resolve => setTimeout(resolve, 2000));
  }

  throw new Error(`Scrape timed out after ${timeoutMs}ms: ${jobId}`);
}

This is less convenient than a single request, but it gives you better control. You can cap waiting time, log failure reasons, retry selectively, and avoid pretending that a challenge page is valid content.

The tradeoff is latency. Browser rendering can take seconds. For interactive coding sessions, you may want a fast path for normal pages and an async rendered path only when the raw fetch looks wrong.

A simple heuristic catches many misses:

function probablyNeedsBrowser(html: string) {
  return html.includes('<div id="root"></div>') ||
    html.includes('<div id="app"></div>') ||
    html.includes('Just a moment...') ||
    html.length < 1000;
}

It is not perfect. It is better than silently summarizing an empty shell.

Structured extraction should validate output

Agents are good at reading pages, but you should not let them invent data contracts on every run. If you need product names, prices, dates, or availability, define the shape and validate it.

import { z } from "zod";

const PricingPlan = z.object({
  name: z.string(),
  monthlyPriceUsd: z.number().nullable(),
  limits: z.array(z.string()),
  sourceUrl: z.string().url()
});

const PricingPlans = z.array(PricingPlan);

const raw = await web.extract<unknown>(markdown, "pricing_plans");
const plans = PricingPlans.parse(raw);

If parsing fails, you have a real error:

ZodError: Expected number, received string at [0].monthlyPriceUsd

That is much easier to debug than a paragraph saying "the pricing appears to be...".

For sites without APIs, selector-based scrapers often break when teams rename classes or move content into a new component. Wire addresses that specific case with pre-built read actions that return typed JSON for supported sites instead of asking the agent to infer structure from HTML.

Coverage is the obvious tradeoff. A catalog action is useful when it exists and stays maintained. A generic scraper still needs to handle everything else.

Expose the tools through MCP if your agent supports it

MCP is a practical way to give different agents the same tool boundary. The exact server can be your own service, a hosted connector, or a thin wrapper around internal APIs.

For example, a Claude Code setup might look like this:

claude mcp add --transport http webdata https://mcp.example.internal/mcp

Cursor uses JSON config instead:

{
  "mcpServers": {
    "webdata": {
      "url": "https://mcp.example.internal/mcp"
    }
  }
}

The important part is not the client. It is that the agent sees separate tools with clear behavior: search, fetchPage, crawl, and extractStructuredData. Do not hide every operation behind one vague "browse the web" function.

A reasonable starting point

Start small. Add search results with snippets, a page fetcher that can detect obvious bad HTML, and schema validation for one workflow you actually care about. Log the source URL, fetch mode, status, content length, and validation errors.

Then run your agent against five pages that have caused problems before: a static doc page, a SPA, a pricing page, a bot-protected page, and a page with stale indexed content. The gaps will show up quickly, and you can decide whether to add rendering, async jobs, crawling, or site-specific structured actions from there.