How to Build an Autonomous AI Web Research Agent with CrewAI & Smart Markdown API

#python #ai #aiagents #rag

When building autonomous AI agents (like CrewAI, AutoGen, or LangGraph) that search and read web documentation, feeding raw HTML causes context truncation errors.

Here is how to equip your CrewAI agent with clean Markdown scraping:

Step 1: Install Dependencies

pip install crewai requests

Step 2: Define the Scraping Tool

import requests
from crewai.tools import tool

@tool("Smart Web Scraper")
def scrape_webpage(url: str) -> str:
    """Scrapes a URL and returns clean Markdown optimized for LLMs."""
    endpoint = "https://smart-markdown-web-scraper.p.rapidapi.com/scrape"
    headers = {
        "x-rapidapi-key": "YOUR_RAPIDAPI_KEY",
        "x-rapidapi-host": "smart-markdown-web-scraper.p.rapidapi.com",
        "Content-Type": "application/json"
    }
    response = requests.post(endpoint, json={"url": url}, headers=headers)
    return response.json().get("markdown", "")