DEV Community

Kafka Queue
Kafka Queue

Posted on

Giving Claude Code and Cursor a web fetch that can read JS-rendered pages

I got tired of my coding agent telling me a page was empty when I could see the content in a browser.

That happens a lot with JavaScript-built sites, cookie banners, and Google results that only finish writing the AI Overview after the shell has loaded. The built-in fetch in Claude Code or Cursor is fine for a static doc. It is not fine when the thing you actually need is painted on the client, or when the site answers a script with a wall.

So I built webfetch.run: a hosted MCP server (and a plain HTTP API) that opens the URL in a real browser, waits, and hands the page back as markdown an agent can read. Failures are free, and the reply says why instead of leaving you to decode an HTTP 200 that is actually a block page.

This is a technical walkthrough of how I use it, not a claim that I solved crawling. I also want to be upfront: as of 16 November 2026, the prices on the site are the intended list, and checkout is not enabled yet. You can sign in with a free account, no card, and get an API key. I have not finished making paid top-ups live.

What it is

One MCP server: https://mcp.webfetch.run/mcp (Streamable HTTP).

Two tools:

  • fetch — the product. Pass a public http(s) URL. You get markdown by default, or any combination of html, text, a screenshot, links, images, metadata, and a few other formats.
  • check_coverage — a free preflight. It only tells you whether the URL is a public http(s) address. It does not fetch the page and it costs nothing.

The same work is available as POST https://api.webfetch.run/v1/scrape if you would rather call HTTP yourself. I reach for MCP in Claude Code and Cursor so I do not have to write the request.

Every fetch first tries a plain HTTP get. If that returns the page, you should not be charged. Credit is meant for the fallback: a browser, a proxy, or a challenge solve. Egress can be direct, datacenter, or residential; the intended price does not change with the tier. If the page comes back empty or blocked, the charge is $0.00 and the response includes a reason (blocked, timeout, and so on).

The MCP fetch tool is a slightly smaller surface than the HTTP API. Today it takes url, formats, only_main_content, egress, and timeout_ms. Things like waitForSelector, session replay, and JSON-schema extraction exist on the HTTP API and are not in the MCP tool schema yet.

It's listed in the official MCP Registry as run.webfetch/webfetch.

Setup

You need an API key from the console. The MCP server does not store a key of its own; it forwards yours. Some clients can also sign in with OAuth. I still use the header, because it is obvious in a config file.

Claude Code

claude mcp add --transport http webfetch \
  https://mcp.webfetch.run/mcp \
  --header "X-API-Key: $WEBFETCH_API_KEY"
Enter fullscreen mode Exit fullscreen mode

Then ask it to fetch a URL and answer from the page, not from memory. A typical prompt I use:

Fetch https://www.google.com/search?q=how+to+run+better as markdown and summarize the AI Overview, People Also Ask, and the top organic results. If the fetch is blocked or empty, tell me the error instead of guessing.

Cursor

In mcp.json (user or project):

{
  "mcpServers": {
    "webfetch": {
      "url": "https://mcp.webfetch.run/mcp",
      "headers": {
        "X-API-Key": "sk_live_YOUR_KEY"
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Claude Desktop uses the same url + headers shape in claude_desktop_config.json.

There is a sandbox with a Google results example preloaded, plus Shopify, Stripe docs, and a plain HTML page. It requires a free sign-in. Signed out, the demo API returns 401 ("Sign in to try a scrape") and the scrape API returns missing_api_key. No card, but it is still a sign-in wall.

Why I bothered

A results page is built for a person looking at a screen. The AI Overview on Google is a good example: it is not in the first HTML paint. Marketing sites hide the actual product copy behind a consent wall. Docs sites ship an empty shell and hydrate later.

None of that is aimed at an agent. I wanted one call that does the boring part — render, retry from another exit if we get a wall, return markdown or a screenshot — and that does not charge me when it failed.

I will not pretend every page comes back. Some sites still block a headless browser. When that happens, webfetch.run should say blocked and cost nothing. The useful next step is usually egress: "residential", not changing the URL or the format.

How this compares, honestly

I did not run a fresh head-to-head bench for this article. Here is how I think about the tools I bump into.

Claude's / Cursor's built-in WebFetch. Use this first. It is already there, no extra account, and it is enough for many static pages. The reason webfetch.run exists is the cases where that built-in fetch comes back thin, stale, or blocked. If the built-in tool works, I leave it alone.

Jina Reader. I like it when I want a very small "URL in, markdown out" step, especially for articles. It is a reader, not a fetch-with-proxies product. I would not assume it will get through a consent wall or a JS-heavy app just because it returned markdown for a blog post. If all you need is readable article text, start there; it is less to set up.

Firecrawl. This is the one people mention first for agent scraping. It is a broader product: crawl, map, extract, and a well-known MCP server. If you need to walk a whole site, use that. webfetch.run is narrower: one URL, one price, a browser and proxies included, and a failed fetch is free. I am not trying to replace a crawler. I am trying to make the single page my agent was asked to read actually show up.

If your job is "get the docs site into a corpus," use a crawler. If your job is "the agent has this URL and needs the rendered page," a single fetch is the whole problem.

Pricing, as it stands

The intended list on the pricing page is $0.022 per successful rendered fetch, $0.00 if no content comes back, with a no-card sign-in and a $25–$1,000 top-up once checkout is on.

Two caveats, so I am not selling a story the site contradicts. The footer still says checkout is not yet enabled. And the free-tier wording disagrees with itself: one block says 100 plain fetches a day; another says plain HTTP pages are free and uncapped, plus 10 rendered fetches free for life, with API access starting at a $25 top-up. The page's structured data describes 100 plain fetches a day and $0.022 for hard pages after a $25 top-up. Until I clean that up, treat the numbers as the intended list, not a reconciled promise.

Enterprise (volume, invoicing, a named contact) is listed. I would rather talk about that directly than put a form here.

Tradeoffs I am not hiding

  • You have to sign in and hold an API key (or complete OAuth). That is more friction than a built-in fetch.
  • It is a hosted service. You are sending the URL, and the rendered content, through my infrastructure. If you need the fetch to stay on your own machines, this is the wrong tool.
  • I do not have a public GitHub repo for the server yet. The official registry entry is remote-only.
  • The MCP tool is not the full HTTP API.
  • A blocked page is still a blocked page. Residential egress helps some walls; it will not help all of them.
  • I have not published a benchmark against Firecrawl or Jina on a fixed URL set. If you try the same pages on two tools, I would like to hear where webfetch.run loses.

If you want to try it

Docs are at webfetch.run/docs. The compact contract I keep in context is elements.txt. Sign in, copy a key, add the MCP server, and fetch a page your agent already failed on. If it comes back empty, the error should be enough to go on. If it isn't, tell me.

Top comments (1)

Collapse
 
li_yi_dcec96bf2647dec1b7e profile image
Mindutil •

The reason field is the part I'd defend hardest, and I have a fresh dataset for why.

Today I ran a liveness pass over 300 of the 4,323 listings in my AI tools directory, one machine, plain fetch, follow redirects, 15s timeout. The status code column turned out to be the least informative one I collected: 84 listings returned no TCP answer at all, and two of those were notion.so and writesonic.com - alive companies that just do not answer from my network path. Another 21 were 403 bot walls on sites that happily serve a browser. So "28% dead" was really "28% undiagnosable from here".

Two cases where I needed exactly what webfetch.run returns: a listing that now redirects to a parked-domain click script answers 200 the whole way, and a page whose

is a widget demo rather than the product also answers 200. Neither is a fetch problem, both are "what actually answered" problems. If your response carried the resolved final host plus the document title, a directory could tell those apart without owning a browser pool.</p>