DEV Community

Cover image for I launched an MCP server on Product Hunt today. Its real user is the agent.
Simon
Simon

Posted on AI-assisted

I launched an MCP server on Product Hunt today. Its real user is the agent.

#showdev #mcp #ai #webscraping

CrawlForge MCP went live on Product Hunt today. It's an open-source MCP server that gives Claude, Cursor and other agents 31 web tools: scrape, search, crawl, structured extraction, stealth browsing, browser sessions that survive between calls, and multi-source research.

Launch posts usually list features. This one is about the thing that changed how I built it: the person reading the docs isn't the user. The agent is. It reads every tool description, on every turn, and it does exactly what they say.


TL;DR: Tool descriptions are your UI, so put cost and "when not to use this" in them. Return markdown, not HTML, because the agent pays for every character. Report a bot wall as a block, never as a successful page. Default to a local model for LLM extraction. CrawlForge is live on Product Hunt today, and I'd love feedback on any of this.

1. Tool descriptions are the interface

In early September I parsed the server's own call log from real agent sessions. 21% of calls repeated an identical call within 30 minutes. The most common chains were search_web → extract_content and fetch_url → extract_metadata. Both fetch the same page twice, when scrape does the job in one call.

The agent wasn't being careless. My descriptions taught it that. fetch_url called itself "ideal as the first step before extract_text", so Claude did what it was told.

Now every description says what the tool is for, what it costs, and which tool to use instead. Here's part of the one scrape ships with today:

Use this to read one page - markdown by default, plus any of "html", "rawHtml",
"text", "links", "metadata", ... all from one fetch. Ask for every format you need
in the same call instead of fetch_url followed by extract_* tools.
...
Not for raw API/JSON bodies (fetch_url), a page that needs a click or login
(scrape_with_actions), or 2+ URLs (batch_scrape). Cost: 2 credits.
Enter fullscreen mode Exit fullscreen mode

Error results work the same way. Each ends with a Next step: line that names the tool to try next, so a failed call steers the agent instead of setting off a retry loop.

If you build MCP tools, the "Not for…" sentence did more for us than any other change. Agents follow an instruction they can see much better than they follow a convention they're supposed to infer.

2. Markdown, because the agent pays per token

Raw HTML is mostly navigation, scripts and attributes the model has to read and then ignore. Wikipedia's "Web scraping" article came to 235,784 characters of HTML and 45,625 characters of CrawlForge markdown, about a fifth of the size, for the same article. Every tool that returns page content defaults to markdown, and scrape takes several formats in one call (markdown, links, metadata, a JSON schema), so the agent asks once.

3. A bot wall is a block, not a page

The worst thing a scraping tool can do to an agent is hand back a Cloudflare challenge page with success: true. The agent then summarises "Checking your browser…" as if it were the article.

So scrape reports a wall as success: false, with blocked.vendor naming Cloudflare, DataDome, PerimeterX, Akamai, Amazon or Vercel, whatever HTTP status came back. With escalate: true, the same call climbs a ladder instead: a plain fetch (2 credits), then a Chrome TLS handshake that still sends our real CrawlForge User-Agent, then a stealth browser (2 + 5). If the wall holds on every rung, the call is charged nothing. robots.txt is honoured by default, and there's no CAPTCHA solving. When a site really doesn't want automated visitors, the honest answer is a clear block.

For flows that need several steps, browser_session keeps one page alive across calls (3 credits to open, then 1–2 per action). The agent can look at the page before it decides what to click, and a login happens once.

4. Local model by default

extract_with_llm, summarize_content, deep_research and scrape's JSON extraction use a local Ollama model by default, with no key and no setup beyond Ollama running. Add an OpenAI or Anthropic key and that provider takes over. Most people trying a new tool don't want to paste a third API key into it first.

Try it

One command adds it to Claude Code (other clients are on the integrations page):

claude mcp add crawlforge -s user \
  -e CRAWLFORGE_API_KEY=cf_live_your_key \
  -- npx -y crawlforge-mcp-server@latest mcp
Enter fullscreen mode Exit fullscreen mode

Run /mcp and you should see crawlforge connected with 31 tools. A free account gets 1,000 credits once, with no card required, and calls start at 1 credit. The server code is MIT on GitHub. Every tool call goes through a CrawlForge API key, including when you run the server yourself.

Feedback welcome

I'm on the Product Hunt thread all day today. I'd most like to hear from anyone who has built MCP tools:

  • Do you put cost and "when not to use" text in your tool descriptions, or does that live somewhere else?
  • What does your agent do when a page comes back walled?
  • Which web task does your agent still get wrong, even with tools?

Answers here in the comments are just as welcome.

Top comments (0)