How to give your AI agent web browsing in 5 minutes (with code)

#ai #webdev #langchain #python

Every AI agent eventually needs to browse the web. Whether it's research, competitor monitoring, or data extraction — at some point your agent needs to read a URL.

The problem? Setting up Puppeteer yourself is a rabbit hole. And the popular scraping APIs charge $19-49/month.

Here's how to wire up web browsing for your agent in under 5 minutes using CrawlAPI (free tier available).

LangChain (Python)

from langchain.tools import Tool
import requests

def scrape_url(url: str) -> str:
    response = requests.post(
        "https://crawlapi.net/v1/scrape",
        headers={"X-RapidAPI-Key": "YOUR_KEY"},
        json={"url": url, "formats": ["markdown"]}
    )
    return response.json()["data"]["markdown"]

web_tool = Tool(
    name="WebScraper",
    func=scrape_url,
    description="Scrape any URL and return clean markdown content. Input should be a URL."
)

# Add to your agent
tools = [web_tool]

CrewAI

from crewai_tools import tool
import requests

@tool("Web Scraper")
def scrape_website(url: str) -> str:
    """Scrape a URL and return its content as clean markdown."""
    response = requests.post(
        "https://crawlapi.net/v1/scrape",
        headers={"X-RapidAPI-Key": "YOUR_KEY"},
        json={"url": url, "formats": ["markdown"]}
    )
    return response.json()["data"]["markdown"]

OpenAI Function Calling

{
  "name": "scrape_url",
  "description": "Scrape a webpage and return its content as clean markdown",
  "parameters": {
    "type": "object",
    "properties": {
      "url": {
        "type": "string",
        "description": "The URL to scrape"
      }
    },
    "required": ["url"]
  }
}

def handle_tool_call(url: str) -> str:
    response = requests.post(
        "https://crawlapi.net/v1/scrape",
        headers={"X-RapidAPI-Key": "YOUR_KEY"},
        json={"url": url, "formats": ["markdown"]}
    )
    return response.json()["data"]["markdown"]

Bonus: Search + scrape in one call

Need your agent to research a topic? One endpoint, no Googling required:

response = requests.post(
    "https://crawlapi.net/v1/search",
    headers={"X-RapidAPI-Key": "YOUR_KEY"},
    json={"query": "latest news on AI agents", "num": 5}
)
# Returns scraped content from top 5 results