DEV Community: Jason Wolf

How to give your AI agent web browsing in 5 minutes (with code)

Jason Wolf — Fri, 10 Apr 2026 10:52:58 +0000

Every AI agent eventually needs to browse the web. Whether it's research, competitor monitoring, or data extraction — at some point your agent needs to read a URL.

The problem? Setting up Puppeteer yourself is a rabbit hole. And the popular scraping APIs charge $19-49/month.

Here's how to wire up web browsing for your agent in under 5 minutes using CrawlAPI (free tier available).

LangChain (Python)

from langchain.tools import Tool
import requests

def scrape_url(url: str) -> str:
    response = requests.post(
        "https://crawlapi.net/v1/scrape",
        headers={"X-RapidAPI-Key": "YOUR_KEY"},
        json={"url": url, "formats": ["markdown"]}
    )
    return response.json()["data"]["markdown"]

web_tool = Tool(
    name="WebScraper",
    func=scrape_url,
    description="Scrape any URL and return clean markdown content. Input should be a URL."
)

# Add to your agent
tools = [web_tool]

CrewAI

from crewai_tools import tool
import requests

@tool("Web Scraper")
def scrape_website(url: str) -> str:
    """Scrape a URL and return its content as clean markdown."""
    response = requests.post(
        "https://crawlapi.net/v1/scrape",
        headers={"X-RapidAPI-Key": "YOUR_KEY"},
        json={"url": url, "formats": ["markdown"]}
    )
    return response.json()["data"]["markdown"]

OpenAI Function Calling

{
  "name": "scrape_url",
  "description": "Scrape a webpage and return its content as clean markdown",
  "parameters": {
    "type": "object",
    "properties": {
      "url": {
        "type": "string",
        "description": "The URL to scrape"
      }
    },
    "required": ["url"]
  }
}

def handle_tool_call(url: str) -> str:
    response = requests.post(
        "https://crawlapi.net/v1/scrape",
        headers={"X-RapidAPI-Key": "YOUR_KEY"},
        json={"url": url, "formats": ["markdown"]}
    )
    return response.json()["data"]["markdown"]

Bonus: Search + scrape in one call

Need your agent to research a topic? One endpoint, no Googling required:

response = requests.post(
    "https://crawlapi.net/v1/search",
    headers={"X-RapidAPI-Key": "YOUR_KEY"},
    json={"query": "latest news on AI agents", "num": 5}
)
# Returns scraped content from top 5 results

Get started

Free tier: 50 calls/day, no credit card
Available on RapidAPI: search "CrawlAPI"
Docs + landing page: crawlapi.net

What are you building? Drop it in the comments — always curious what people are using agents for.

Add web scraping to Claude Desktop in 60 seconds (MCP + CrawlAPI)

Jason Wolf — Fri, 10 Apr 2026 10:52:37 +0000

Claude Desktop is powerful. But by default it can't browse the web.

Here's how to fix that in 60 seconds using the CrawlAPI MCP server — no coding required.

What you get

Three tools available directly in Claude (and any MCP-compatible editor like Cursor or Windsurf):

scrape_url — fetch any page as clean markdown
batch_scrape — scrape up to 10 URLs in parallel
search_and_scrape — search the web + scrape top results in one shot

Setup

1. Get a free API key

Head to RapidAPI and subscribe (free tier: 50 calls/day, no credit card).

2. Download the MCP server

git clone https://github.com/JDWolf96/crawlapi-js
cd crawlapi-js
npm install

3. Add to Claude Desktop config

Open ~/Library/Application Support/Claude/claude_desktop_config.json (Mac) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "crawlapi": {
      "command": "node",
      "args": ["/absolute/path/to/crawlapi-js/mcp-server.js"],
      "env": {
        "CRAWLAPI_KEY": "your_rapidapi_key_here"
      }
    }
  }
}

4. Restart Claude Desktop

That's it. You'll see the CrawlAPI tools appear in the tools panel.

What it looks like in practice

Ask Claude:

"Scrape https://news.ycombinator.com and summarise the top 5 stories"

"Search for 'LangChain vs LlamaIndex 2026' and give me a comparison"

"Batch scrape these 5 competitor pages and tell me how they position themselves"

Claude uses the tools automatically — you just ask in plain English.

Cursor / Windsurf / Continue

Same config, different location. Check your editor's MCP settings panel. The server works with any stdio-transport MCP client.

Why markdown matters

Most scraping APIs return raw HTML. CrawlAPI returns clean markdown — no nav bars, no cookie banners, no scripts. Just the content, ready to drop into an LLM context window.

It also handles JavaScript rendering, so pages that load content dynamically (SPAs, React apps, etc.) work out of the box.

Best Web Scraping APIs for AI Agents in 2026 (Compared)

Jason Wolf — Wed, 08 Apr 2026 17:21:09 +0000

If you're building AI agents in 2026, you'll hit this problem within the first week: your agent needs to read a webpage, and suddenly you're deep in a Puppeteer rabbit hole, debugging headless Chrome, and wondering why a simple task takes three days.

Web scraping APIs exist to solve this. But which one is actually worth using for an AI agent use case?

I compared the four main options: Firecrawl, ScrapingBee, Apify, and CrawlAPI. Here's the honest breakdown.

───

The Contenders

Firecrawl — The current darling of the AI agent world. Clean markdown output, good docs. Starting price: $19/month for 500 credits.

ScrapingBee — The enterprise option. Handles proxies, CAPTCHAs, JS rendering. Starting price: $49/month.

Apify — A full scraping platform. Starting price: $49/month. Powerful but complex.

CrawlAPI — Newer, built specifically for AI agents. Clean markdown, Puppeteer fallback, Redis caching. Starting price: $9.99/month for 5,000 calls. Free tier: 50 calls/day, no credit card.

───

Which one should you use?

Prototyping / side project: CrawlAPI. Free tier is genuinely useful, $9.99/mo when you upgrade, copy-paste manifests for LangChain + CrewAI + OpenAI Assistants.

Production, high volume: Firecrawl if you want established, CrawlAPI if cost matters.

Enterprise (proxy rotation, CAPTCHA): ScrapingBee or Apify.

Using CrewAI: CrawlAPI — only one with a native tool manifest.

───

LangChain integration (CrawlAPI)

from langchain.tools import Tool
import requests

def scrape_url(url: str) -> str:
r = requests.post(
"https://crawlapi.net/v1/scrape",
headers={"X-RapidAPI-Key": "YOUR_KEY"},
json={"url": url, "formats": ["markdown"]}
)
return r.json()["data"]["markdown"]

web_tool = Tool(
name="WebScraper",
func=scrape_url,
description="Scrape any URL and return clean markdown. Input should be a URL."
)

CrewAI integration (CrawlAPI)

from crewai_tools import tool
import requests

@tool("Web Scraper")
def scrape_website(url: str) -> str:
"""Scrape a URL and return its content as clean markdown."""
r = requests.post(
"https://crawlapi.net/v1/scrape",
headers={"X-RapidAPI-Key": "YOUR_KEY"},
json={"url": url, "formats": ["markdown"]}
)
return r.json()["data"]["markdown"]

───

CrawlAPI is the cheapest option that still checks all the boxes for agent use cases. Try it free: crawlapi.net

Have a different tool you'd recommend? Drop it in the comments.